[==========] Running 38 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 37 tests from ExternalMiniClusterTest
[ RUN ] ExternalMiniClusterTest.TestKerberosReacquire
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:52:42 dist-test-slave-p3t5 krb5kdc[25664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:52:42 dist-test-slave-p3t5 krb5kdc[25664](info): set up 2 sockets
Aug 12 19:52:42 dist-test-slave-p3t5 krb5kdc[25664](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 19:52:44.662098 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.184s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:52:44 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:52:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:52:44.950892 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33873
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:33873 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:52:45.310863 25687 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:52:45.311252 25687 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:52:45.311450 25687 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:52:45.320245 25687 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:52:45.320369 25687 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:52:45.320483 25687 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:52:45.320564 25687 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:52:45.332032 25687 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33873
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:52:45.333895 25687 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:52:45.335462 25687 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:52:45.350277 25692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:52:45.350677 25696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:52:45.350301 25694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:52:45.352151 25687 server_base.cc:1061] running on GCE node
I20260812 19:52:45.353026 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:52:45.354655 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:52:45.355943 25687 hybrid_clock.cc:648] HybridClock initialized: now 1786564365355883 us; error 92 us; skew 500 ppm
I20260812 19:52:45.356388 25687 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:52:45 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564365, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:52:45.359884 25687 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:52:45.361753 25687 webserver.cc:533] Webserver started at http://127.25.14.62:33767/ using document root <none> and password file <none>
I20260812 19:52:45.362579 25687 fs_manager.cc:362] Metadata directory not provided
I20260812 19:52:45.362725 25687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:52:45.363125 25687 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:52:45.365962 25687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "f819dd008d73489fa6cfceab588b8564"
format_stamp: "Formatted at 2026-08-12 19:52:45 on dist-test-slave-p3t5"
I20260812 19:52:45.367233 25687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "f819dd008d73489fa6cfceab588b8564"
format_stamp: "Formatted at 2026-08-12 19:52:45 on dist-test-slave-p3t5"
I20260812 19:52:45.373229 25687 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:52:45.377017 25704 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:52:45.378750 25687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:52:45.378968 25687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "f819dd008d73489fa6cfceab588b8564"
format_stamp: "Formatted at 2026-08-12 19:52:45 on dist-test-slave-p3t5"
I20260812 19:52:45.379247 25687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:52:45.412004 25687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:52:45.420495 25687 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:52:45.420863 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:52:45.444525 25687 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33873
I20260812 19:52:45.444545 25764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33873 every 8 connection(s)
I20260812 19:52:45.446460 25687 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:52:45.452322 25765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:52:45.455957 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25687
I20260812 19:52:45.456302 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:52:45.464676 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Bootstrap starting.
I20260812 19:52:45.468546 25765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Neither blocks nor log segments found. Creating new log.
I20260812 19:52:45.469877 25765 log.cc:826] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Log is configured to *not* fsync() on all Append() calls
I20260812 19:52:45.473096 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: No bootstrap required, opened a new log
I20260812 19:52:45.479667 25765 raft_consensus.cc:359] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:52:45.480269 25765 raft_consensus.cc:385] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:52:45.480436 25765 raft_consensus.cc:740] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819dd008d73489fa6cfceab588b8564, State: Initialized, Role: FOLLOWER
I20260812 19:52:45.481176 25765 consensus_queue.cc:260] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:52:45.481439 25765 raft_consensus.cc:399] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 19:52:45.481690 25765 raft_consensus.cc:493] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 19:52:45.481913 25765 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:52:45.484540 25765 raft_consensus.cc:515] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
Aug 12 19:52:45 dist-test-slave-p3t5 krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:52:45.485320 25765 leader_election.cc:304] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f819dd008d73489fa6cfceab588b8564; no voters:
I20260812 19:52:45.486197 25765 leader_election.cc:290] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 19:52:45.486342 25772 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:52:45.487037 25772 raft_consensus.cc:697] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 LEADER]: Becoming Leader. State: Replica: f819dd008d73489fa6cfceab588b8564, State: Running, Role: LEADER
I20260812 19:52:45.487895 25772 consensus_queue.cc:237] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:52:45.492350 25765 sys_catalog.cc:565] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:52:45.492218 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f819dd008d73489fa6cfceab588b8564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } } }
I20260812 19:52:45.492617 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: This master's current role is: LEADER
I20260812 19:52:45.493330 25773 sys_catalog.cc:455] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f819dd008d73489fa6cfceab588b8564. Latest consensus state: current_term: 1 leader_uuid: "f819dd008d73489fa6cfceab588b8564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } } }
I20260812 19:52:45.493662 25773 sys_catalog.cc:458] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: This master's current role is: LEADER
I20260812 19:52:45.495975 25777 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:52:45.508994 25777 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:52:45.521330 25777 catalog_manager.cc:1383] Generated new cluster ID: 3cefd6825fca47868215a4c647b84988
I20260812 19:52:45.521579 25777 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:52:45.548586 25777 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:52:45.550038 25777 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:52:45.564195 25777 catalog_manager.cc:6092] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Generated new TSK 0
I20260812 19:52:45.565224 25777 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:52:45.701803 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33873
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:52:46.088827 25799 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:52:46.089190 25799 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:52:46.089286 25799 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:52:46.098366 25799 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:52:46.098569 25799 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:52:46.109865 25799 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:52:46.111598 25799 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:52:46.113122 25799 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:52:46.123654 25810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:52:46.123917 25807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:52:46.124575 25812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:52:46.125347 25799 server_base.cc:1061] running on GCE node
I20260812 19:52:46.126119 25799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:52:46.127144 25799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:52:46.128400 25799 hybrid_clock.cc:648] HybridClock initialized: now 1786564366128325 us; error 66 us; skew 500 ppm
I20260812 19:52:46.128741 25799 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:52:46 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564366, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:52:46.132200 25799 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:52:46.134109 25799 webserver.cc:533] Webserver started at http://127.25.14.1:34639/ using document root <none> and password file <none>
I20260812 19:52:46.134912 25799 fs_manager.cc:362] Metadata directory not provided
I20260812 19:52:46.135058 25799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:52:46.135462 25799 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:52:46.138548 25799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "52318066961e4265bf926e2d8458a0d5"
format_stamp: "Formatted at 2026-08-12 19:52:46 on dist-test-slave-p3t5"
I20260812 19:52:46.139624 25799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "52318066961e4265bf926e2d8458a0d5"
format_stamp: "Formatted at 2026-08-12 19:52:46 on dist-test-slave-p3t5"
I20260812 19:52:46.145819 25799 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 19:52:46.149544 25820 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:52:46.151085 25799 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:52:46.151386 25799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "52318066961e4265bf926e2d8458a0d5"
format_stamp: "Formatted at 2026-08-12 19:52:46 on dist-test-slave-p3t5"
I20260812 19:52:46.151607 25799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:52:46.190164 25799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:52:46.194383 25799 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:52:46.194710 25799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:52:46.195786 25799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:52:46.197510 25799 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:52:46.197649 25799 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:52:46.197816 25799 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:52:46.197893 25799 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:52:46.235442 25799 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39263
I20260812 19:52:46.235502 25952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39263 every 8 connection(s)
I20260812 19:52:46.237524 25799 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:52:46.247908 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25799
I20260812 19:52:46.248167 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:52:46 dist-test-slave-p3t5 krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564366, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:52:46.260756 25954 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33873
I20260812 19:52:46.261197 25954 heartbeater.cc:461] Registering TS with master...
I20260812 19:52:46.262274 25954 heartbeater.cc:507] Master 127.25.14.62:33873 requested a full tablet report, sending...
I20260812 19:52:46.264940 25725 ts_manager.cc:194] Registered new tserver with Master: 52318066961e4265bf926e2d8458a0d5 (127.25.14.1:39263)
I20260812 19:52:46.266803 25725 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:37993
I20260812 19:52:46.267176 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:52:47.270920 25954 heartbeater.cc:499] Master 127.25.14.62:33873 was elected leader, sending a full tablet report...
Aug 12 19:53:00 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564380, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:53:00.362386 25700 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 19:53:01 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564381, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:53:01.134624 25815 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 19:53:02 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260812 19:53:02.307931 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25687
I20260812 19:53:02.309731 25759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:02.419392 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25687
I20260812 19:53:02.451122 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33873
--webserver_interface=127.25.14.62
--webserver_port=33767
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:33873 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:53:02.820483 25961 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:02.820924 25961 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:02.821092 25961 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:02.830662 25961 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:02.830842 25961 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:02.830991 25961 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:02.831079 25961 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:02.842609 25961 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33873
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=33767
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:02.844506 25961 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:02.846051 25961 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:02.855647 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:02.855739 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:02.855765 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:02.856499 25961 server_base.cc:1061] running on GCE node
I20260812 19:53:02.857518 25961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:02.859163 25961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:02.860428 25961 hybrid_clock.cc:648] HybridClock initialized: now 1786564382860387 us; error 38 us; skew 500 ppm
I20260812 19:53:02.860829 25961 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:53:02 dist-test-slave-p3t5 krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564382, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:53:02.864552 25961 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:53:02.866487 25961 webserver.cc:533] Webserver started at http://127.25.14.62:33767/ using document root <none> and password file <none>
I20260812 19:53:02.867321 25961 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:02.867501 25961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:02.873533 25961 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:53:02.876796 25977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:02.878432 25961 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:02.878643 25961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "f819dd008d73489fa6cfceab588b8564"
format_stamp: "Formatted at 2026-08-12 19:52:45 on dist-test-slave-p3t5"
I20260812 19:53:02.879442 25961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:02.893009 25961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:02.897394 25961 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:02.897809 25961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:02.920049 25961 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33873
I20260812 19:53:02.920075 26035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33873 every 8 connection(s)
I20260812 19:53:02.921911 25961 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:02.925596 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25961
I20260812 19:53:02.931135 26036 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:02.934867 26036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Bootstrap starting.
Aug 12 19:53:02 dist-test-slave-p3t5 krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:53:02.957460 26036 log.cc:826] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:02.967283 26036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:02.968137 26036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Bootstrap complete.
I20260812 19:53:02.975049 26036 raft_consensus.cc:359] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:53:02.975857 26036 raft_consensus.cc:740] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819dd008d73489fa6cfceab588b8564, State: Initialized, Role: FOLLOWER
I20260812 19:53:02.976670 26036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:53:02.976925 26036 raft_consensus.cc:399] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 19:53:02.977065 26036 raft_consensus.cc:493] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 19:53:02.977283 26036 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:02.981999 26036 raft_consensus.cc:515] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:53:02.982625 26036 leader_election.cc:304] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f819dd008d73489fa6cfceab588b8564; no voters:
I20260812 19:53:02.983314 26036 leader_election.cc:290] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 19:53:02.983546 26041 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:53:02.984270 26041 raft_consensus.cc:697] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [term 2 LEADER]: Becoming Leader. State: Replica: f819dd008d73489fa6cfceab588b8564, State: Running, Role: LEADER
I20260812 19:53:02.985013 26041 consensus_queue.cc:237] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } }
I20260812 19:53:02.985762 26036 sys_catalog.cc:565] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:02.987908 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f819dd008d73489fa6cfceab588b8564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } } }
I20260812 19:53:02.988435 26043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f819dd008d73489fa6cfceab588b8564. Latest consensus state: current_term: 2 leader_uuid: "f819dd008d73489fa6cfceab588b8564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f819dd008d73489fa6cfceab588b8564" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33873 } } }
I20260812 19:53:02.988710 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:02.988839 26043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:02.991140 26049 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:02.998956 26049 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:03.000660 26049 catalog_manager.cc:1295] Loaded cluster ID: 3cefd6825fca47868215a4c647b84988
I20260812 19:53:03.000802 26049 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:03.004920 26049 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:03.006690 26049 catalog_manager.cc:6103] T 00000000000000000000000000000000 P f819dd008d73489fa6cfceab588b8564: Loaded TSK: 0
I20260812 19:53:03.008610 26049 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:03.321358 25996 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "52318066961e4265bf926e2d8458a0d5" instance_seqno: 1786564366226446) as {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:50329; Asking this server to re-register.
I20260812 19:53:03.322446 25954 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:03.322790 25954 heartbeater.cc:507] Master 127.25.14.62:33873 requested a full tablet report, sending...
I20260812 19:53:03.324491 25996 ts_manager.cc:194] Registered new tserver with Master: 52318066961e4265bf926e2d8458a0d5 (127.25.14.1:39263)
I20260812 19:53:03.327862 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:53:03.328154 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25799
Aug 12 19:53:03 dist-test-slave-p3t5 krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:53:03.348176 25948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:03.440168 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25799
I20260812 19:53:03.474466 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25961
I20260812 19:53:03.475802 26031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:03.583839 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25961
2026-08-12T19:53:03Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestKerberosReacquire (21173 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/0
I20260812 19:53:03.638365 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
W20260812 19:53:04.004158 26069 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:04.004611 26069 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:04.004756 26069 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:04.014099 26069 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:04.014246 26069 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:04.014360 26069 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:04.014449 26069 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:04.026607 26069 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:04.028518 26069 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:04.030575 26069 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:04.039803 26074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:04.039820 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:04.040597 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:04.041412 26069 server_base.cc:1061] running on GCE node
I20260812 19:53:04.042448 26069 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:04.042671 26069 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:04.042793 26069 hybrid_clock.cc:648] HybridClock initialized: now 1786564384042766 us; error 0 us; skew 500 ppm
I20260812 19:53:04.043203 26069 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:04.046718 26069 webserver.cc:533] Webserver started at http://127.25.14.62:43139/ using document root <none> and password file <none>
I20260812 19:53:04.047688 26069 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:04.047961 26069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:04.048492 26069 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:04.051291 26069 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.052222 26069 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.058599 26069 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 19:53:04.062819 26085 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:04.064527 26069 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:53:04.064945 26069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.065284 26069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:04.078869 26069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:04.079977 26069 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:04.080291 26069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:04.102994 26069 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:39485
I20260812 19:53:04.103015 26141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:39485 every 8 connection(s)
I20260812 19:53:04.104835 26069 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:04.109769 26142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:04.113261 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26069
I20260812 19:53:04.113539 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:04.114809 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:33935
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
I20260812 19:53:04.116146 26142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:04.139689 26142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:04.140982 26088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
W20260812 19:53:04.142731 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
I20260812 19:53:04.192149 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } attempt: 1
W20260812 19:53:04.194726 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
I20260812 19:53:04.263056 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } attempt: 2
W20260812 19:53:04.265342 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
I20260812 19:53:04.356669 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } attempt: 3
W20260812 19:53:04.359346 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
W20260812 19:53:04.492493 26149 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:04.492846 26149 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:04.493000 26149 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:04.501590 26149 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:04.501724 26149 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:04.501845 26149 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:04.501916 26149 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:04.502650 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } attempt: 4
W20260812 19:53:04.504856 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111)
I20260812 19:53:04.513223 26149 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:04.515112 26149 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:04.517112 26149 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:04.526075 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:04.526834 26159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:04.527123 26157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:04.527794 26149 server_base.cc:1061] running on GCE node
I20260812 19:53:04.528818 26149 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:04.529026 26149 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:04.529135 26149 hybrid_clock.cc:648] HybridClock initialized: now 1786564384529111 us; error 0 us; skew 500 ppm
I20260812 19:53:04.529523 26149 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:04.533567 26149 webserver.cc:533] Webserver started at http://127.25.14.61:36281/ using document root <none> and password file <none>
I20260812 19:53:04.534489 26149 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:04.534592 26149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:04.534950 26149 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:04.537576 26149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "8f113a4c427946d3a7e1313ff17f8d77"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.538575 26149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "8f113a4c427946d3a7e1313ff17f8d77"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.544930 26149 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:53:04.548928 26164 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:04.550729 26149 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:04.550984 26149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "8f113a4c427946d3a7e1313ff17f8d77"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:04.551296 26149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:04.572753 26149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:04.573904 26149 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:04.574210 26149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:04.596444 26149 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:33935
I20260812 19:53:04.596452 26227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:33935 every 8 connection(s)
I20260812 19:53:04.598328 26149 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:04.600123 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26149
I20260812 19:53:04.600337 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:04.602012 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43195
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
I20260812 19:53:04.604952 26229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:04.613157 26229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:04.628166 26229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:04.638083 26229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:04.639463 26167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
W20260812 19:53:04.641280 26229 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.690623 26229 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 1
W20260812 19:53:04.692926 26229 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.761263 26229 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 2
W20260812 19:53:04.763767 26229 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.804207 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } attempt: 5
I20260812 19:53:04.811825 26142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:04.814298 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.855074 26229 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 3
W20260812 19:53:04.857492 26229 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.865583 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 1
W20260812 19:53:04.868146 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:04.936420 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 2
W20260812 19:53:04.938962 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
W20260812 19:53:04.974984 26231 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:04.975340 26231 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:04.975533 26231 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:04.984062 26231 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:04.984174 26231 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:04.984273 26231 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:04.984354 26231 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:04.994742 26231 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:04.996594 26231 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:04.998663 26231 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:05.000828 26229 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 4
W20260812 19:53:05.003139 26229 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
W20260812 19:53:05.008136 26240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:05.008836 26242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:05.009153 26244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:05.009799 26231 server_base.cc:1061] running on GCE node
I20260812 19:53:05.010660 26231 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:05.010841 26231 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:05.010958 26231 hybrid_clock.cc:648] HybridClock initialized: now 1786564385010917 us; error 0 us; skew 500 ppm
I20260812 19:53:05.011330 26231 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:05.014430 26231 webserver.cc:533] Webserver started at http://127.25.14.60:38983/ using document root <none> and password file <none>
I20260812 19:53:05.015242 26231 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:05.015378 26231 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:05.015769 26231 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:05.018356 26231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "827d1eca26d54393b66f0bd67574d30e"
format_stamp: "Formatted at 2026-08-12 19:53:05 on dist-test-slave-p3t5"
I20260812 19:53:05.019202 26231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "827d1eca26d54393b66f0bd67574d30e"
format_stamp: "Formatted at 2026-08-12 19:53:05 on dist-test-slave-p3t5"
I20260812 19:53:05.024863 26231 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 19:53:05.028496 26250 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:05.030129 26231 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:05.030324 26231 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "827d1eca26d54393b66f0bd67574d30e"
format_stamp: "Formatted at 2026-08-12 19:53:05 on dist-test-slave-p3t5"
I20260812 19:53:05.030593 26231 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:05.045249 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 3
W20260812 19:53:05.047685 26142 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111)
I20260812 19:53:05.048813 26231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:05.049934 26231 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:05.050243 26231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:05.073916 26231 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:43195
I20260812 19:53:05.073936 26320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:43195 every 8 connection(s)
I20260812 19:53:05.075688 26231 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:05.077843 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26231
I20260812 19:53:05.078101 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:05.081674 26321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:05.090042 26321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:05.102505 26321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:05.110684 26321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:05.121603 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Bootstrap starting.
I20260812 19:53:05.125041 26321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:05.126485 26321 log.cc:826] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:05.129465 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: No bootstrap required, opened a new log
I20260812 19:53:05.134871 26321 raft_consensus.cc:359] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.135210 26321 raft_consensus.cc:385] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:05.135337 26321 raft_consensus.cc:740] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827d1eca26d54393b66f0bd67574d30e, State: Initialized, Role: FOLLOWER
I20260812 19:53:05.136061 26321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.137271 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.137678 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.138688 26321 sys_catalog.cc:565] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:53:05.149501 26341 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:05.149678 26341 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:53:05.146603 26231 master_runner.cc:187] Error getting master registration for 127.25.14.62:39485: OK, instance_id { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" instance_seqno: 1786564384096831 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:05.152208 26231 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:05.225003 26142 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 4
I20260812 19:53:05.238471 26142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap starting.
I20260812 19:53:05.243088 26142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:05.244601 26142 log.cc:826] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:05.248207 26142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: No bootstrap required, opened a new log
I20260812 19:53:05.253732 26142 raft_consensus.cc:359] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.254096 26142 raft_consensus.cc:385] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:05.254235 26142 raft_consensus.cc:740] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60635e5a875425ebc26296a3e50c687, State: Initialized, Role: FOLLOWER
I20260812 19:53:05.254913 26142 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.255997 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.256328 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.257278 26142 sys_catalog.cc:565] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:05.268168 26327 raft_consensus.cc:493] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:05.268497 26327 raft_consensus.cc:515] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
W20260812 19:53:05.270362 26358 catalog_manager.cc:1594] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:05.270586 26358 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:53:05.270519 26327 leader_election.cc:290] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485), 8f113a4c427946d3a7e1313ff17f8d77 (127.25.14.61:33935)
I20260812 19:53:05.271723 26115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827d1eca26d54393b66f0bd67574d30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60635e5a875425ebc26296a3e50c687" is_pre_election: true
I20260812 19:53:05.272368 26115 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827d1eca26d54393b66f0bd67574d30e in term 0.
I20260812 19:53:05.273190 26254 leader_election.cc:304] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827d1eca26d54393b66f0bd67574d30e, e60635e5a875425ebc26296a3e50c687; no voters:
I20260812 19:53:05.273720 26327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:53:05.273871 26327 raft_consensus.cc:493] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:05.273988 26327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:05.277673 26327 raft_consensus.cc:515] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.279080 26115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827d1eca26d54393b66f0bd67574d30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60635e5a875425ebc26296a3e50c687"
I20260812 19:53:05.279338 26115 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:05.272258 26069 master_runner.cc:187] Error getting master registration for 127.25.14.61:33935: OK, instance_id { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" instance_seqno: 1786564384590553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:05.280975 26069 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:05.282379 26327 leader_election.cc:290] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [CANDIDATE]: Term 1 election: Requested vote from peers e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485), 8f113a4c427946d3a7e1313ff17f8d77 (127.25.14.61:33935)
I20260812 19:53:05.283214 26115 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827d1eca26d54393b66f0bd67574d30e in term 1.
I20260812 19:53:05.283794 26254 leader_election.cc:304] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827d1eca26d54393b66f0bd67574d30e, e60635e5a875425ebc26296a3e50c687; no voters:
I20260812 19:53:05.284199 26327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:53:05.284940 26327 raft_consensus.cc:697] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 LEADER]: Becoming Leader. State: Replica: 827d1eca26d54393b66f0bd67574d30e, State: Running, Role: LEADER
I20260812 19:53:05.285597 26327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.300616 26360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827d1eca26d54393b66f0bd67574d30e. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.301003 26360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:05.302372 26229 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } attempt: 5
W20260812 19:53:05.302580 26257 tablet.cc:2406] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:53:05.304920 26362 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:05.310837 26362 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:05.317128 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Bootstrap starting.
I20260812 19:53:05.323351 26229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:05.324507 26229 log.cc:826] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Log is configured to *not* fsync() on all Append() calls
W20260812 19:53:05.333554 26091 tablet.cc:2406] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:53:05.335848 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: No bootstrap required, opened a new log
I20260812 19:53:05.344542 26229 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.351179 26115 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Refusing update from remote peer 827d1eca26d54393b66f0bd67574d30e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:53:05.350180 26229 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:05.352077 26229 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f113a4c427946d3a7e1313ff17f8d77, State: Initialized, Role: FOLLOWER
I20260812 19:53:05.352324 26360 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:05.354252 26229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:05.363642 26229 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:05.361477 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.365700 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.367765 26196 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:05.369962 26198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827d1eca26d54393b66f0bd67574d30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f113a4c427946d3a7e1313ff17f8d77"
I20260812 19:53:05.371994 26199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827d1eca26d54393b66f0bd67574d30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f113a4c427946d3a7e1313ff17f8d77" is_pre_election: true
I20260812 19:53:05.375288 26196 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Refusing update from remote peer 827d1eca26d54393b66f0bd67574d30e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:53:05.376314 26360 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:05.395262 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827d1eca26d54393b66f0bd67574d30e. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.395597 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.399178 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827d1eca26d54393b66f0bd67574d30e. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.399446 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.399684 26364 mvcc.cc:204] Tried to move back new op lower bound from 7317767722384502784 to 7317767722136440832. Current Snapshot: MvccSnapshot[applied={T|T < 7317767722384502784}]
I20260812 19:53:05.407821 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.408084 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.434885 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.435226 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:05.435732 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.435957 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:05.440896 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:05.441170 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:05.443590 26362 catalog_manager.cc:1383] Generated new cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:05.443722 26362 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:05.448400 26387 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 19:53:05.470288 26362 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:53:05.472359 26362 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:05.496040 26362 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Generated new TSK 0
I20260812 19:53:05.497329 26362 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:05.535488 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
W20260812 19:53:05.982405 26390 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:05.982771 26390 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:05.982849 26390 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:05.995515 26390 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:05.995769 26390 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:06.011699 26390 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:06.013967 26390 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:06.016786 26390 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:06.030748 26395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:06.031498 26396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:06.031934 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:06.033331 26390 server_base.cc:1061] running on GCE node
I20260812 19:53:06.034204 26390 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:06.034561 26390 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:06.034821 26390 hybrid_clock.cc:648] HybridClock initialized: now 1786564386034639 us; error 0 us; skew 500 ppm
I20260812 19:53:06.035357 26390 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:06.038483 26390 webserver.cc:533] Webserver started at http://127.25.14.1:46443/ using document root <none> and password file <none>
I20260812 19:53:06.039494 26390 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:06.039690 26390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:06.040290 26390 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:06.043660 26390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.044759 26390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.051484 26390 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 19:53:06.055985 26403 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:06.058135 26390 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:06.058439 26390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.058935 26390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:06.080641 26390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:06.081910 26390 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:06.082252 26390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:06.083488 26390 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:06.085652 26390 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:06.085789 26390 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 19:53:06.085924 26390 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:06.086035 26390 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:06.129434 26390 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38009
I20260812 19:53:06.129477 26539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38009 every 8 connection(s)
I20260812 19:53:06.131189 26390 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:06.133246 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26390
I20260812 19:53:06.133509 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:06.140956 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
I20260812 19:53:06.179205 26543 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:06.179406 26541 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:06.179714 26543 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.180943 26543 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:06.179260 26540 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:06.181332 26540 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.179716 26541 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.181975 26540 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:06.182402 26541 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:06.184155 26188 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:06.184144 26276 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:06.184211 26102 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:06.186427 26276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41045
I20260812 19:53:06.272735 26358 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:06.272850 26358 catalog_manager.cc:1588] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: loading cluster ID for follower catalog manager: success
I20260812 19:53:06.275804 26358 catalog_manager.cc:1610] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: acquiring CA information for follower catalog manager: success
I20260812 19:53:06.278394 26358 catalog_manager.cc:1638] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:06.454818 26387 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:06.454973 26387 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: loading cluster ID for follower catalog manager: success
I20260812 19:53:06.459702 26387 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: acquiring CA information for follower catalog manager: success
I20260812 19:53:06.463092 26387 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:53:06.618003 26553 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:06.618386 26553 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:06.618547 26553 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:06.627985 26553 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:06.628261 26553 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:06.640969 26553 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:06.643208 26553 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:06.645953 26553 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:06.657531 26559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:06.658635 26560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:06.664505 26562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:06.670572 26553 server_base.cc:1061] running on GCE node
I20260812 19:53:06.671483 26553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:06.671700 26553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:06.671820 26553 hybrid_clock.cc:648] HybridClock initialized: now 1786564386671788 us; error 0 us; skew 500 ppm
I20260812 19:53:06.672231 26553 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:06.675208 26553 webserver.cc:533] Webserver started at http://127.25.14.2:45319/ using document root <none> and password file <none>
I20260812 19:53:06.676169 26553 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:06.676328 26553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:06.676800 26553 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:06.680110 26553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "d76925d829a74feea231c6ad2a9a4d8b"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.681041 26553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "d76925d829a74feea231c6ad2a9a4d8b"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.688122 26553 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 19:53:06.692507 26567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:06.694300 26553 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:06.694518 26553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "d76925d829a74feea231c6ad2a9a4d8b"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:06.694775 26553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:06.714020 26553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:06.715081 26553 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:06.715437 26553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:06.716575 26553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:06.718506 26553 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:06.718639 26553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:06.718768 26553 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:06.718861 26553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:06.759745 26553 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:44549
I20260812 19:53:06.759801 26701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:44549 every 8 connection(s)
I20260812 19:53:06.761667 26553 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:06.767179 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26553
I20260812 19:53:06.767458 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:06.773173 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
I20260812 19:53:06.796708 26702 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:06.797122 26704 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:06.797247 26702 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.797303 26704 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.798820 26702 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:06.799274 26703 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:06.799443 26703 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:06.800108 26703 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:06.800381 26704 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:06.801335 26276 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:44549)
I20260812 19:53:06.801442 26102 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:44549)
I20260812 19:53:06.801558 26188 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:44549)
I20260812 19:53:06.802601 26276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:43623
I20260812 19:53:07.189839 26540 heartbeater.cc:499] Master 127.25.14.60:43195 was elected leader, sending a full tablet report...
W20260812 19:53:07.214015 26712 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:07.215358 26712 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:07.215521 26712 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:07.225448 26712 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:07.225745 26712 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:07.237450 26712 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:07.239481 26712 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:07.241887 26712 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:07.254678 26718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:07.254714 26720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:07.254643 26717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:07.255551 26712 server_base.cc:1061] running on GCE node
I20260812 19:53:07.256276 26712 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:07.256486 26712 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:07.256597 26712 hybrid_clock.cc:648] HybridClock initialized: now 1786564387256571 us; error 0 us; skew 500 ppm
I20260812 19:53:07.257061 26712 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:07.259835 26712 webserver.cc:533] Webserver started at http://127.25.14.3:43031/ using document root <none> and password file <none>
I20260812 19:53:07.260644 26712 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:07.260766 26712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:07.261188 26712 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:07.264178 26712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "281416abd38d474f93144da1177c79de"
format_stamp: "Formatted at 2026-08-12 19:53:07 on dist-test-slave-p3t5"
I20260812 19:53:07.265296 26712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "281416abd38d474f93144da1177c79de"
format_stamp: "Formatted at 2026-08-12 19:53:07 on dist-test-slave-p3t5"
I20260812 19:53:07.272014 26712 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260812 19:53:07.276437 26725 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:07.278404 26712 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:53:07.278676 26712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "281416abd38d474f93144da1177c79de"
format_stamp: "Formatted at 2026-08-12 19:53:07 on dist-test-slave-p3t5"
I20260812 19:53:07.278964 26712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:07.298833 26712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:07.300001 26712 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:07.300326 26712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:07.301601 26712 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:07.303728 26712 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:07.303859 26712 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:07.303948 26712 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:07.304008 26712 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:07.347661 26712 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:38857
I20260812 19:53:07.347711 26856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:38857 every 8 connection(s)
I20260812 19:53:07.349360 26712 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:07.355966 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26712
I20260812 19:53:07.356168 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:07.383201 26858 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:07.383708 26858 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:07.384006 26857 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:07.384162 26857 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:07.384970 26858 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:07.384989 26857 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:07.385445 26859 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:07.385989 26859 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:07.386581 26859 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:07.387219 26275 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:38857)
I20260812 19:53:07.387418 26102 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:38857)
I20260812 19:53:07.388262 26188 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:38857)
I20260812 19:53:07.388398 26275 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:45255
I20260812 19:53:07.404284 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:07.414397 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:07.414686 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26069
I20260812 19:53:07.415856 26136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:07.530227 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26069
I20260812 19:53:07.561602 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--webserver_interface=127.25.14.62
--webserver_port=43139
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
W20260812 19:53:07.597493 26254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111)
W20260812 19:53:07.599252 26254 consensus_peers.cc:597] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e -> Peer e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485): Couldn't send request to peer e60635e5a875425ebc26296a3e50c687. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:53:07.805217 26702 heartbeater.cc:499] Master 127.25.14.60:43195 was elected leader, sending a full tablet report...
W20260812 19:53:07.805744 26703 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:39485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111)
W20260812 19:53:07.980836 26869 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:07.981241 26869 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:07.981323 26869 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:07.990381 26869 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:07.990499 26869 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:07.990561 26869 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:07.990607 26869 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:08.001703 26869 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=43139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:08.003512 26869 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:08.005439 26869 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:08.015910 26881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:08.016791 26883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:08.016824 26880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:08.018324 26869 server_base.cc:1061] running on GCE node
I20260812 19:53:08.019351 26869 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:08.019539 26869 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:08.019618 26869 hybrid_clock.cc:648] HybridClock initialized: now 1786564388019599 us; error 0 us; skew 500 ppm
I20260812 19:53:08.020056 26869 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:08.023372 26869 webserver.cc:533] Webserver started at http://127.25.14.62:43139/ using document root <none> and password file <none>
I20260812 19:53:08.024241 26869 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:08.024367 26869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:08.030764 26869 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:53:08.034152 26888 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:08.035898 26869 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:53:08.036139 26869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:08.037125 26869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:08.067504 26869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:08.068563 26869 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:08.068918 26869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:08.085026 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:08.085407 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:08.094893 26869 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:39485
I20260812 19:53:08.094990 26947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:39485 every 8 connection(s)
I20260812 19:53:08.097021 26869 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:08.098987 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26869
I20260812 19:53:08.099419 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26390
I20260812 19:53:08.107084 26949 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:08.109959 26533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:08.111908 26949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap starting.
I20260812 19:53:08.131592 26949 log.cc:826] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:08.142491 26949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:08.143278 26949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap complete.
I20260812 19:53:08.151104 26949 raft_consensus.cc:359] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:08.151971 26949 raft_consensus.cc:740] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60635e5a875425ebc26296a3e50c687, State: Initialized, Role: FOLLOWER
I20260812 19:53:08.152858 26949 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:08.153950 26957 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:08.154325 26957 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:08.155287 26949 sys_catalog.cc:565] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:08.167725 26970 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:08.167877 26970 catalog_manager.cc:1588] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: loading cluster ID for follower catalog manager: success
I20260812 19:53:08.173856 26970 catalog_manager.cc:1610] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: acquiring CA information for follower catalog manager: success
I20260812 19:53:08.177294 26970 catalog_manager.cc:1638] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:08.234736 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26390
W20260812 19:53:08.262396 26973 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:53:08.232374 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:60795 (local address 127.25.14.62:39485)
0812 19:53:08.233333 (+ 959us) server_negotiation.cc:207] Beginning negotiation
0812 19:53:08.233339 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:53:08.233356 (+ 17us) server_negotiation.cc:408] Connection header received
0812 19:53:08.233432 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:53:08.233438 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:53:08.233529 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:53:08.233659 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0812 19:53:08.234687 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:53:08.236121 (+ 1434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:53:08.261142 (+ 25021us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:60795: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":798}
I20260812 19:53:08.262781 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38009
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=46443
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
I20260812 19:53:08.391297 26857 heartbeater.cc:499] Master 127.25.14.60:43195 was elected leader, sending a full tablet report...
I20260812 19:53:08.405431 26906 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "281416abd38d474f93144da1177c79de" instance_seqno: 1786564387338264) as {username='slave'} at 127.25.14.3:46857; Asking this server to re-register.
I20260812 19:53:08.406549 26858 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:08.408957 26906 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:38857)
I20260812 19:53:08.689388 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827d1eca26d54393b66f0bd67574d30e. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:08.689764 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:08.745683 26978 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:08.746238 26978 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:08.746407 26978 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:08.755595 26978 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:08.755863 26978 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:08.767374 26978 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46443
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:08.769254 26978 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:08.771335 26978 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:08.783209 26993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:08.783125 26991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:08.783183 26995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:08.783929 26978 server_base.cc:1061] running on GCE node
I20260812 19:53:08.784672 26978 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:08.784914 26978 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:08.785307 26978 hybrid_clock.cc:648] HybridClock initialized: now 1786564388785273 us; error 0 us; skew 500 ppm
I20260812 19:53:08.785791 26978 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:08.788579 26978 webserver.cc:533] Webserver started at http://127.25.14.1:46443/ using document root <none> and password file <none>
I20260812 19:53:08.789562 26978 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:08.789790 26978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:08.796573 26978 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 19:53:08.800475 27001 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:08.802474 26978 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:08.802677 26978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:08.803473 26978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:08.814452 26703 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:08.815805 26906 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d76925d829a74feea231c6ad2a9a4d8b" instance_seqno: 1786564386750049) as {username='slave'} at 127.25.14.2:41573; Asking this server to re-register.
I20260812 19:53:08.816447 26703 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:08.817757 26906 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:44549)
I20260812 19:53:08.823503 26978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:08.824807 26978 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:08.825169 26978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:08.826431 26978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:08.828606 26978 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:08.828686 26978 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:08.828862 26978 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:08.828928 26978 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:08.869414 27137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38009 every 8 connection(s)
I20260812 19:53:08.869516 26978 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38009
I20260812 19:53:08.871251 26978 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:08.876837 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26978
I20260812 19:53:08.877351 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26978
I20260812 19:53:08.901221 27138 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:08.901466 27140 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:08.901810 27140 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:08.901808 27138 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:08.903071 27139 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:08.903261 27139 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:08.903313 27138 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:08.903390 27140 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:08.903947 27139 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:08.905704 26275 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:08.905864 26188 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:08.906252 26906 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:08.906946 26275 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:36259
I20260812 19:53:08.909454 27132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:09.004719 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26978
I20260812 19:53:09.033043 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38009
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=46443
--webserver_interface=127.25.14.1
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260812 19:53:09.247817 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827d1eca26d54393b66f0bd67574d30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:09.248267 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: LEADER
W20260812 19:53:09.515748 27153 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:09.516126 27153 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:09.516209 27153 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:09.516274 27153 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:09.524971 27153 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:09.525177 27153 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:09.536322 27153 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46443
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:09.538373 27153 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:09.540648 27153 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:09.552845 27162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:09.552743 27163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:09.553305 27167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:09.553673 27153 server_base.cc:1061] running on GCE node
I20260812 19:53:09.554427 27153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:09.554606 27153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:09.554746 27153 hybrid_clock.cc:648] HybridClock initialized: now 1786564389554703 us; error 0 us; skew 500 ppm
I20260812 19:53:09.555106 27153 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:09.557886 27153 webserver.cc:533] Webserver started at http://127.25.14.1:46443/ using document root <none> and password file <none>
I20260812 19:53:09.558773 27153 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:09.558930 27153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:09.565698 27153 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 19:53:09.569043 27172 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:09.571033 27153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 19:53:09.571277 27153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:09.572201 27153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:09.597065 27153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:09.598223 27153 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:09.598543 27153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:09.599694 27153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:09.601709 27153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:09.601851 27153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:09.602002 27153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:09.602097 27153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:09.602579 27153 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:09.602690 27153 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:09.602766 27153 fault_injection.cc:59] injecting fault for kudu (pid 27153): FLAGS_fault_before_start (process will exit)
I20260812 19:53:09.632409 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38009
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=46443
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
W20260812 19:53:10.011507 27180 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:10.011937 27180 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:10.012091 27180 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:10.021190 27180 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:10.021474 27180 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:10.032037 27180 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46443
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:10.033964 27180 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:10.036027 27180 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:10.047025 27188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:10.047066 27186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:10.046864 27185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:10.048310 27180 server_base.cc:1061] running on GCE node
I20260812 19:53:10.048938 27180 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:10.049101 27180 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:10.049185 27180 hybrid_clock.cc:648] HybridClock initialized: now 1786564390049164 us; error 0 us; skew 500 ppm
I20260812 19:53:10.049484 27180 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:10.052016 27180 webserver.cc:533] Webserver started at http://127.25.14.1:46443/ using document root <none> and password file <none>
I20260812 19:53:10.052856 27180 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:10.052969 27180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:10.059667 27180 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 19:53:10.062856 27195 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:10.064703 27180 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:10.064946 27180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:10.065845 27180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:10.107632 27180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:10.108688 27180 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:10.109035 27180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:10.110169 27180 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:10.112196 27180 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:10.112321 27180 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:10.112497 27180 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:10.112581 27180 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:10.149560 27180 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38009
I20260812 19:53:10.149600 27318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38009 every 8 connection(s)
I20260812 19:53:10.151262 27180 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:10.159773 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27180
I20260812 19:53:10.160302 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27180
I20260812 19:53:10.226912 27321 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:10.226939 27320 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:10.227485 27320 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:10.227486 27321 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:10.228744 27321 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:10.228780 27320 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:10.229494 27314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:10.230901 26188 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:10.231091 26906 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:10.318732 27319 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:10.318964 27319 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:10.319654 27319 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:10.320312 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27180
I20260812 19:53:10.321004 26275 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:38009)
I20260812 19:53:10.322171 26275 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:45331
W20260812 19:53:10.346304 26254 connection.cc:573] server connection from 127.25.14.1:45331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:10.346642 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26553
I20260812 19:53:10.355906 26697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:10.446826 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26553
I20260812 19:53:10.474331 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26712
I20260812 19:53:10.483355 26851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:10.575771 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26712
I20260812 19:53:10.600817 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26869
I20260812 19:53:10.607865 26943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:10.716514 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26869
I20260812 19:53:10.748708 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26149
I20260812 19:53:10.750201 26222 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 19:53:10.775707 26254 consensus_peers.cc:597] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e -> Peer e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485): Couldn't send request to peer e60635e5a875425ebc26296a3e50c687. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:53:10.863389 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26149
W20260812 19:53:10.891641 26253 connection.cc:573] client connection to 127.25.14.61:33935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:10.891769 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26231
W20260812 19:53:10.893137 26253 consensus_peers.cc:597] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e -> Peer 8f113a4c427946d3a7e1313ff17f8d77 (127.25.14.61:33935): Couldn't send request to peer 8f113a4c427946d3a7e1313ff17f8d77. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:53:10.893204 26313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:11.004890 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26231
I20260812 19:53:11.040962 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
W20260812 19:53:11.408427 27345 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:11.408885 27345 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:11.409029 27345 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:11.417691 27345 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:11.417851 27345 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:11.417963 27345 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:11.418051 27345 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:11.428472 27345 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:11.430434 27345 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:11.432528 27345 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:11.441663 27352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:11.442219 27351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:11.443039 27354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:11.443483 27345 server_base.cc:1061] running on GCE node
I20260812 19:53:11.444315 27345 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:11.444536 27345 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:11.444638 27345 hybrid_clock.cc:648] HybridClock initialized: now 1786564391444614 us; error 0 us; skew 500 ppm
I20260812 19:53:11.445009 27345 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:11.448242 27345 webserver.cc:533] Webserver started at http://127.25.14.62:33021/ using document root <none> and password file <none>
I20260812 19:53:11.449154 27345 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:11.449287 27345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:11.455224 27345 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:53:11.458607 27361 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:11.460246 27345 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:11.460471 27345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:11.461279 27345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:11.492767 27345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:11.493886 27345 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:11.494221 27345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:11.516736 27345 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:39485
I20260812 19:53:11.516775 27423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:39485 every 8 connection(s)
I20260812 19:53:11.518535 27345 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:11.526247 27424 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:11.526762 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27345
I20260812 19:53:11.526974 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:11.528460 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:33935
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
I20260812 19:53:11.531579 27424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap starting.
I20260812 19:53:11.551512 27424 log.cc:826] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:11.561271 27424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:11.561930 27424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap complete.
I20260812 19:53:11.569545 27424 raft_consensus.cc:359] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:11.570443 27424 raft_consensus.cc:740] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60635e5a875425ebc26296a3e50c687, State: Initialized, Role: FOLLOWER
I20260812 19:53:11.571363 27424 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:11.572388 27429 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:11.572723 27429 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:11.573589 27424 sys_catalog.cc:565] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:11.585677 27442 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:11.585945 27442 catalog_manager.cc:1588] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: loading cluster ID for follower catalog manager: success
I20260812 19:53:11.591943 27442 catalog_manager.cc:1610] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: acquiring CA information for follower catalog manager: success
I20260812 19:53:11.598546 27442 catalog_manager.cc:1638] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:11.611073 27345 master_runner.cc:187] Error getting master registration for 127.25.14.61:33935: Network error: Client connection negotiation failed: client connection to 127.25.14.61:33935: connect: Connection refused (error 111),
I20260812 19:53:11.615053 27345 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:11.929493 27427 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:11.929943 27427 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:11.930109 27427 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:11.938885 27427 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:11.938990 27427 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:11.939085 27427 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:11.939162 27427 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:11.950127 27427 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:11.952005 27427 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:11.954137 27427 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:11.963016 27452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:11.963016 27455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:11.964543 27458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:11.965368 27427 server_base.cc:1061] running on GCE node
I20260812 19:53:11.966313 27427 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:11.966528 27427 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:11.966635 27427 hybrid_clock.cc:648] HybridClock initialized: now 1786564391966611 us; error 0 us; skew 500 ppm
I20260812 19:53:11.967079 27427 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:11.970255 27427 webserver.cc:533] Webserver started at http://127.25.14.61:35395/ using document root <none> and password file <none>
I20260812 19:53:11.971097 27427 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:11.971266 27427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:11.978116 27427 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:53:11.982198 27463 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:11.984256 27427 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:11.984470 27427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "8f113a4c427946d3a7e1313ff17f8d77"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:11.985668 27427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:12.011585 27427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:12.012866 27427 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:12.013218 27427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:12.036285 27427 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:33935
I20260812 19:53:12.036327 27518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:33935 every 8 connection(s)
I20260812 19:53:12.038132 27427 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:12.045156 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27427
I20260812 19:53:12.045399 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:12.046206 27519 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:12.046705 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43195
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
I20260812 19:53:12.051946 27519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Bootstrap starting.
I20260812 19:53:12.070905 27519 log.cc:826] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:12.081373 27519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:12.082033 27519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Bootstrap complete.
I20260812 19:53:12.089376 27519 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:12.090332 27519 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f113a4c427946d3a7e1313ff17f8d77, State: Initialized, Role: FOLLOWER
I20260812 19:53:12.091174 27519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:12.092257 27524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:12.092612 27524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:12.093561 27519 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:12.105897 27536 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:12.107255 27536 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: loading cluster ID for follower catalog manager: success
I20260812 19:53:12.111904 27536 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: acquiring CA information for follower catalog manager: success
I20260812 19:53:12.114919 27536 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:12.134441 27427 master_runner.cc:187] Error getting master registration for 127.25.14.60:43195: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43195: connect: Connection refused (error 111),
I20260812 19:53:12.138309 27427 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:12.468132 27522 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:12.468495 27522 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:12.468662 27522 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:12.477707 27522 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:12.477816 27522 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:12.477909 27522 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:12.477996 27522 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:12.488451 27522 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:12.490294 27522 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:12.492288 27522 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:12.501461 27544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:12.502525 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:12.503350 27548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:12.504065 27522 server_base.cc:1061] running on GCE node
I20260812 19:53:12.504994 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:12.505204 27522 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:12.505309 27522 hybrid_clock.cc:648] HybridClock initialized: now 1786564392505287 us; error 0 us; skew 500 ppm
I20260812 19:53:12.505790 27522 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:12.508775 27522 webserver.cc:533] Webserver started at http://127.25.14.60:36527/ using document root <none> and password file <none>
I20260812 19:53:12.509603 27522 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:12.509802 27522 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:12.516043 27522 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:12.520246 27556 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:12.522207 27522 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:12.522493 27522 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "827d1eca26d54393b66f0bd67574d30e"
format_stamp: "Formatted at 2026-08-12 19:53:05 on dist-test-slave-p3t5"
I20260812 19:53:12.523573 27522 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:12.552258 27522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:12.553292 27522 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:12.553649 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:12.578858 27522 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:43195
I20260812 19:53:12.578918 27616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:43195 every 8 connection(s)
I20260812 19:53:12.580760 27522 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:12.585295 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27522
I20260812 19:53:12.585493 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:12.589835 27617 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:12.594726 27617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Bootstrap starting.
I20260812 19:53:12.617955 27617 log.cc:826] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:12.629576 27617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:12.630427 27617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: Bootstrap complete.
I20260812 19:53:12.638782 27617 raft_consensus.cc:359] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:12.639619 27617 raft_consensus.cc:740] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827d1eca26d54393b66f0bd67574d30e, State: Initialized, Role: FOLLOWER
I20260812 19:53:12.640426 27617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:12.641395 27623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:12.641747 27623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:12.642699 27617 sys_catalog.cc:565] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:12.655668 27636 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:12.655915 27636 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: loading cluster ID for follower catalog manager: success
I20260812 19:53:12.661471 27636 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: acquiring CA information for follower catalog manager: success
I20260812 19:53:12.664876 27636 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:12.688469 27522 master_runner.cc:264] No leader master found from master 827d1eca26d54393b66f0bd67574d30e
I20260812 19:53:12.688668 27522 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:13.237109 27639 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:13.237516 27639 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:13.239681 27639 leader_election.cc:290] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485), 827d1eca26d54393b66f0bd67574d30e (127.25.14.60:43195)
I20260812 19:53:13.240630 27393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f113a4c427946d3a7e1313ff17f8d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e60635e5a875425ebc26296a3e50c687" is_pre_election: true
I20260812 19:53:13.241189 27393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f113a4c427946d3a7e1313ff17f8d77 in term 1.
I20260812 19:53:13.241930 27467 leader_election.cc:304] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f113a4c427946d3a7e1313ff17f8d77, e60635e5a875425ebc26296a3e50c687; no voters:
I20260812 19:53:13.242354 27639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:53:13.242524 27639 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:13.242627 27639 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:13.247659 27639 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:13.247619 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f113a4c427946d3a7e1313ff17f8d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "827d1eca26d54393b66f0bd67574d30e" is_pre_election: true
I20260812 19:53:13.248220 27588 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f113a4c427946d3a7e1313ff17f8d77 in term 1.
I20260812 19:53:13.248538 27639 leader_election.cc:290] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [CANDIDATE]: Term 2 election: Requested vote from peers e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485), 827d1eca26d54393b66f0bd67574d30e (127.25.14.60:43195)
I20260812 19:53:13.249361 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f113a4c427946d3a7e1313ff17f8d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "827d1eca26d54393b66f0bd67574d30e"
I20260812 19:53:13.249379 27393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f113a4c427946d3a7e1313ff17f8d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e60635e5a875425ebc26296a3e50c687"
I20260812 19:53:13.249558 27588 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:13.249594 27393 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:13.253449 27588 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f113a4c427946d3a7e1313ff17f8d77 in term 2.
I20260812 19:53:13.253459 27393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f113a4c427946d3a7e1313ff17f8d77 in term 2.
I20260812 19:53:13.254040 27464 leader_election.cc:304] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827d1eca26d54393b66f0bd67574d30e, 8f113a4c427946d3a7e1313ff17f8d77; no voters:
I20260812 19:53:13.254446 27639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:53:13.254858 27639 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [term 2 LEADER]: Becoming Leader. State: Replica: 8f113a4c427946d3a7e1313ff17f8d77, State: Running, Role: LEADER
I20260812 19:53:13.255371 27639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:13.258241 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f113a4c427946d3a7e1313ff17f8d77. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.258482 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:13.259150 27644 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:13.261303 27644 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:13.262208 27644 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:13.262324 27644 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:13.263531 27644 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:13.264364 27644 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77: Loaded TSK: 0
I20260812 19:53:13.265305 27644 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:13.291298 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
W20260812 19:53:13.668808 27645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:13.669171 27645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:13.669317 27645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:13.678283 27645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:13.678494 27645 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:13.688946 27645 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:13.690806 27645 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:13.692860 27645 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:13.696523 27393 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 2 FOLLOWER]: Refusing update from remote peer 8f113a4c427946d3a7e1313ff17f8d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:53:13.697831 27642 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:13.703466 27649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f113a4c427946d3a7e1313ff17f8d77. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.703814 27649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:13.705063 27639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.705366 27639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:13.707118 27588 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [term 2 FOLLOWER]: Refusing update from remote peer 8f113a4c427946d3a7e1313ff17f8d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260812 19:53:13.707087 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:13.708074 27642 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:13.709470 27649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.709755 27649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:13.714047 27657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:13.715234 27661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:13.715337 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.715720 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:13.716140 27658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f113a4c427946d3a7e1313ff17f8d77. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:13.716323 27645 server_base.cc:1061] running on GCE node
I20260812 19:53:13.716497 27658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827d1eca26d54393b66f0bd67574d30e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:13.717075 27645 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:13.717275 27645 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:13.717367 27645 hybrid_clock.cc:648] HybridClock initialized: now 1786564393717348 us; error 0 us; skew 500 ppm
I20260812 19:53:13.717839 27645 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:13.720275 27645 webserver.cc:533] Webserver started at http://127.25.14.1:41617/ using document root <none> and password file <none>
I20260812 19:53:13.721179 27645 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:13.721318 27645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:13.727334 27645 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:53:13.730404 27668 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:13.731989 27645 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:13.732172 27645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:13.732928 27645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:13.751823 27645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:13.752856 27645 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:13.753173 27645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:13.754293 27645 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:13.756179 27645 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:13.756259 27645 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:13.756464 27645 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:13.756525 27645 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:13.794479 27645 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36551
I20260812 19:53:13.794590 27801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36551 every 8 connection(s)
I20260812 19:53:13.796264 27645 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:13.798435 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27645
I20260812 19:53:13.798700 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:13.807928 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
I20260812 19:53:13.828990 27804 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:13.829314 27803 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:13.829473 27804 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:13.829475 27803 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:13.829574 27802 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:13.829900 27802 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:13.830914 27803 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:13.831032 27804 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:13.831067 27802 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:13.833847 27379 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:13.833846 27574 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:13.833848 27481 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:13.836508 27481 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:51903
W20260812 19:53:14.189402 27813 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:14.189858 27813 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:14.189952 27813 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:14.198783 27813 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:14.198956 27813 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:14.209540 27813 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:14.211531 27813 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:14.213491 27813 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:14.224808 27819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:14.224812 27820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:14.225196 27822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:14.225967 27813 server_base.cc:1061] running on GCE node
I20260812 19:53:14.226639 27813 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:14.226857 27813 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:14.226980 27813 hybrid_clock.cc:648] HybridClock initialized: now 1786564394226954 us; error 0 us; skew 500 ppm
I20260812 19:53:14.227344 27813 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:14.229918 27813 webserver.cc:533] Webserver started at http://127.25.14.2:38591/ using document root <none> and password file <none>
I20260812 19:53:14.230804 27813 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:14.230980 27813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:14.237007 27813 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 19:53:14.240180 27828 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.241881 27813 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:14.242098 27813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "d76925d829a74feea231c6ad2a9a4d8b"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:14.242888 27813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:14.259027 27813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:14.260026 27813 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:14.260350 27813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:14.261401 27813 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:14.263288 27813 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:14.263398 27813 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.263554 27813 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:14.263628 27813 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.273504 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:14.273892 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:14.307920 27813 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:34867
I20260812 19:53:14.307971 27958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:34867 every 8 connection(s)
I20260812 19:53:14.309707 27813 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:14.310997 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27813
I20260812 19:53:14.311211 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:14.318107 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--time_source=system_unsync with env {}
I20260812 19:53:14.333465 27959 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:14.333984 27959 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.335032 27960 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:14.335217 27960 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.335302 27959 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:14.335738 27960 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:14.337348 27379 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:34867)
I20260812 19:53:14.337399 27574 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:34867)
I20260812 19:53:14.343140 27961 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:14.343338 27961 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.343822 27961 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:14.344926 27481 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:34867)
I20260812 19:53:14.345875 27481 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:41019
W20260812 19:53:14.715058 27966 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:14.715442 27966 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:14.715618 27966 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:14.724375 27966 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:14.724601 27966 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:14.735351 27966 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:14.737166 27966 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:14.739596 27966 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:14.750506 27971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:14.750926 27972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:14.751377 27974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:14.752698 27966 server_base.cc:1061] running on GCE node
I20260812 19:53:14.753432 27966 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:14.753669 27966 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:14.753813 27966 hybrid_clock.cc:648] HybridClock initialized: now 1786564394753784 us; error 0 us; skew 500 ppm
I20260812 19:53:14.754179 27966 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:14.756562 27966 webserver.cc:533] Webserver started at http://127.25.14.3:35345/ using document root <none> and password file <none>
I20260812 19:53:14.757440 27966 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:14.757686 27966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:14.763950 27966 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:14.767242 27979 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.768886 27966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:14.769059 27966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "281416abd38d474f93144da1177c79de"
format_stamp: "Formatted at 2026-08-12 19:53:07 on dist-test-slave-p3t5"
I20260812 19:53:14.769922 27966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:14.800264 27966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:14.801544 27966 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:14.801931 27966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:14.802991 27966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:14.804786 27966 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:14.804875 27966 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.804984 27966 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:14.805028 27966 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:14.840236 27804 heartbeater.cc:499] Master 127.25.14.61:33935 was elected leader, sending a full tablet report...
I20260812 19:53:14.844834 27966 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:46389
I20260812 19:53:14.844867 28105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:46389 every 8 connection(s)
I20260812 19:53:14.846829 27966 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:14.852690 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27966
I20260812 19:53:14.852910 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:14.874768 28107 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:14.874815 28106 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:14.874874 28108 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:14.875317 28108 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.875314 28106 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.875314 28107 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:14.876449 28107 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:14.876490 28108 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:14.876549 28106 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:14.878418 27481 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:46389)
I20260812 19:53:14.878612 27379 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:46389)
I20260812 19:53:14.878647 27574 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:46389)
I20260812 19:53:14.879616 27481 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:32833
I20260812 19:53:14.890388 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:14.898952 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:14.899180 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27345
I20260812 19:53:14.900156 27419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:15.009660 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27345
I20260812 19:53:15.041102 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--webserver_interface=127.25.14.62
--webserver_port=33021
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195 with env {}
W20260812 19:53:15.327239 27467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111)
W20260812 19:53:15.328913 27467 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 -> Peer e60635e5a875425ebc26296a3e50c687 (127.25.14.62:39485): Couldn't send request to peer e60635e5a875425ebc26296a3e50c687. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:53:15.340597 27960 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:39485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:39485: connect: Connection refused (error 111)
I20260812 19:53:15.348554 27961 heartbeater.cc:499] Master 127.25.14.61:33935 was elected leader, sending a full tablet report...
W20260812 19:53:15.417245 28117 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:15.417737 28117 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:15.417898 28117 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:15.426863 28117 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:15.427018 28117 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:15.427138 28117 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:15.427229 28117 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:15.438342 28117 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:15.440300 28117 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:15.442415 28117 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:15.451889 28127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:15.452348 28126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:15.453050 28129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:15.453416 28117 server_base.cc:1061] running on GCE node
I20260812 19:53:15.454401 28117 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:15.454614 28117 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:15.454723 28117 hybrid_clock.cc:648] HybridClock initialized: now 1786564395454700 us; error 0 us; skew 500 ppm
I20260812 19:53:15.455056 28117 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:15.458488 28117 webserver.cc:533] Webserver started at http://127.25.14.62:33021/ using document root <none> and password file <none>
I20260812 19:53:15.459317 28117 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:15.459446 28117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:15.465544 28117 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:53:15.468789 28134 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:15.470408 28117 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:15.470633 28117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "e60635e5a875425ebc26296a3e50c687"
format_stamp: "Formatted at 2026-08-12 19:53:04 on dist-test-slave-p3t5"
I20260812 19:53:15.471396 28117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:15.491007 28117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:15.492054 28117 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:15.492389 28117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:15.514540 28189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:39485 every 8 connection(s)
I20260812 19:53:15.514508 28117 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:39485
I20260812 19:53:15.516575 28117 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:15.524811 28190 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:15.526572 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28117
I20260812 19:53:15.526964 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27645
I20260812 19:53:15.529670 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap starting.
I20260812 19:53:15.542166 27797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:15.554075 28190 log.cc:826] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:15.565011 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:15.565753 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: Bootstrap complete.
I20260812 19:53:15.573788 28190 raft_consensus.cc:359] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:15.574623 28190 raft_consensus.cc:740] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60635e5a875425ebc26296a3e50c687, State: Initialized, Role: FOLLOWER
I20260812 19:53:15.575591 28190 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } }
I20260812 19:53:15.576623 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:15.576964 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:15.578122 28190 sys_catalog.cc:565] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:15.598142 28210 catalog_manager.cc:1295] Loaded cluster ID: 058d19d9a67e45a6b07999b1adc666cb
I20260812 19:53:15.598397 28210 catalog_manager.cc:1588] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: loading cluster ID for follower catalog manager: success
I20260812 19:53:15.603355 28210 catalog_manager.cc:1610] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: acquiring CA information for follower catalog manager: success
I20260812 19:53:15.606664 28210 catalog_manager.cc:1638] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:15.649159 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27645
I20260812 19:53:15.674436 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36551
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=41617
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
I20260812 19:53:15.720288 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:15.720579 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:15.732527 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f113a4c427946d3a7e1313ff17f8d77. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:15.732852 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60635e5a875425ebc26296a3e50c687 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:15.881868 28108 heartbeater.cc:499] Master 127.25.14.61:33935 was elected leader, sending a full tablet report...
I20260812 19:53:15.894188 28153 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "281416abd38d474f93144da1177c79de" instance_seqno: 1786564394833953) as {username='slave'} at 127.25.14.3:57175; Asking this server to re-register.
I20260812 19:53:15.895252 28107 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:15.897394 28153 ts_manager.cc:194] Registered new tserver with Master: 281416abd38d474f93144da1177c79de (127.25.14.3:46389)
W20260812 19:53:16.074442 28217 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:16.074904 28217 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:16.075070 28217 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:16.084072 28217 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:16.084357 28217 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:16.095229 28217 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=41617
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:16.097088 28217 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:16.099291 28217 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:16.110827 28228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:16.111069 28227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:16.111114 28230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:16.112229 28217 server_base.cc:1061] running on GCE node
I20260812 19:53:16.112897 28217 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:16.113068 28217 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:16.113142 28217 hybrid_clock.cc:648] HybridClock initialized: now 1786564396113122 us; error 0 us; skew 500 ppm
I20260812 19:53:16.113503 28217 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:16.115981 28217 webserver.cc:533] Webserver started at http://127.25.14.1:41617/ using document root <none> and password file <none>
I20260812 19:53:16.116794 28217 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:16.116902 28217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:16.123323 28217 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:53:16.126381 28236 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:16.128053 28217 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:16.128274 28217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:16.129091 28217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:16.152316 28217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:16.153349 28217 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:16.153723 28217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:16.154838 28217 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:16.156770 28217 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:16.156845 28217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:16.156996 28217 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:16.157045 28217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:16.193765 28217 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36551
I20260812 19:53:16.193779 28362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36551 every 8 connection(s)
I20260812 19:53:16.195683 28217 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:16.200361 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28217
I20260812 19:53:16.200759 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28217
I20260812 19:53:16.219935 28366 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:16.221516 28366 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:16.222200 28365 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:16.222414 28365 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:16.223097 28366 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:16.224192 28365 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:16.225396 28153 ts_manager.cc:194] Registered new tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:16.225401 27481 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:16.225699 28363 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:16.225826 28363 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:16.226372 28363 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:16.226665 27481 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:58849
I20260812 19:53:16.227878 27574 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:16.231189 28357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:16.264293 28378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f113a4c427946d3a7e1313ff17f8d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60635e5a875425ebc26296a3e50c687" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 39485 } } peers { permanent_uuid: "8f113a4c427946d3a7e1313ff17f8d77" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 33935 } } peers { permanent_uuid: "827d1eca26d54393b66f0bd67574d30e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43195 } } }
I20260812 19:53:16.264587 28378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f113a4c427946d3a7e1313ff17f8d77 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:16.321516 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28217
I20260812 19:53:16.348217 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36551
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=41617
--webserver_interface=127.25.14.1
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260812 19:53:16.350313 27960 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:16.351567 28153 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d76925d829a74feea231c6ad2a9a4d8b" instance_seqno: 1786564394298802) as {username='slave'} at 127.25.14.2:48863; Asking this server to re-register.
I20260812 19:53:16.352461 27960 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:16.353665 28153 ts_manager.cc:194] Registered new tserver with Master: d76925d829a74feea231c6ad2a9a4d8b (127.25.14.2:34867)
W20260812 19:53:16.721441 28383 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:16.721943 28383 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:16.722050 28383 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:16.722110 28383 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:16.730942 28383 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:16.731158 28383 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:16.741603 28383 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=41617
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:16.743547 28383 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:16.745662 28383 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:16.756759 28388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:16.756646 28390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:16.757743 28392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:16.758199 28383 server_base.cc:1061] running on GCE node
I20260812 19:53:16.758920 28383 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:16.759087 28383 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:16.759219 28383 hybrid_clock.cc:648] HybridClock initialized: now 1786564396759158 us; error 0 us; skew 500 ppm
I20260812 19:53:16.759581 28383 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:16.762238 28383 webserver.cc:533] Webserver started at http://127.25.14.1:41617/ using document root <none> and password file <none>
I20260812 19:53:16.763195 28383 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:16.763372 28383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:16.770135 28383 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:53:16.773478 28398 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:16.775305 28383 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:16.775564 28383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:16.776458 28383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:16.798877 28383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:16.799875 28383 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:16.800185 28383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:16.801330 28383 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:16.803359 28383 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:16.803488 28383 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:16.803653 28383 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:16.803752 28383 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:16.804167 28383 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:16.804315 28383 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:16.804414 28383 fault_injection.cc:59] injecting fault for kudu (pid 28383): FLAGS_fault_before_start (process will exit)
I20260812 19:53:16.835162 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36551
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--webserver_port=41617
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
W20260812 19:53:17.209981 28409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:17.210335 28409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:17.210517 28409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:17.219281 28409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:17.219518 28409 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:17.230266 28409 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=41617
--tserver_master_addrs=127.25.14.62:39485,127.25.14.61:33935,127.25.14.60:43195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:17.232177 28409 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:17.234270 28409 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:17.244976 28415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:17.245039 28414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:17.245743 28417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:17.246631 28409 server_base.cc:1061] running on GCE node
I20260812 19:53:17.247339 28409 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:53:17.247560 28409 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:53:17.247670 28409 hybrid_clock.cc:648] HybridClock initialized: now 1786564397247647 us; error 0 us; skew 500 ppm
I20260812 19:53:17.248095 28409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:17.250701 28409 webserver.cc:533] Webserver started at http://127.25.14.1:41617/ using document root <none> and password file <none>
I20260812 19:53:17.251539 28409 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:17.251649 28409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:17.258589 28409 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:17.261816 28423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:17.263558 28409 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:53:17.263794 28409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "89ce8a89a51c4ffa8375b18e6d8cda86"
format_stamp: "Formatted at 2026-08-12 19:53:06 on dist-test-slave-p3t5"
I20260812 19:53:17.264658 28409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:17.279627 28409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:17.280828 28409 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:17.281126 28409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:17.282347 28409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:17.284468 28409 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:17.284556 28409 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:17.284657 28409 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:17.284706 28409 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:17.322677 28409 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36551
I20260812 19:53:17.322716 28550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36551 every 8 connection(s)
I20260812 19:53:17.324375 28409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:17.332417 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28409
I20260812 19:53:17.332962 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28409
I20260812 19:53:17.358376 28552 heartbeater.cc:344] Connected to a master server at 127.25.14.62:39485
I20260812 19:53:17.359174 28552 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:17.360601 28552 heartbeater.cc:507] Master 127.25.14.62:39485 requested a full tablet report, sending...
I20260812 19:53:17.362886 28153 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:17.365342 28551 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43195
I20260812 19:53:17.365383 28553 heartbeater.cc:344] Connected to a master server at 127.25.14.61:33935
I20260812 19:53:17.365494 28551 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:17.365518 28553 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:17.366103 28551 heartbeater.cc:507] Master 127.25.14.60:43195 requested a full tablet report, sending...
I20260812 19:53:17.366156 28553 heartbeater.cc:507] Master 127.25.14.61:33935 requested a full tablet report, sending...
I20260812 19:53:17.367506 27574 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:17.368113 28546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:17.368207 27481 ts_manager.cc:194] Re-registered known tserver with Master: 89ce8a89a51c4ffa8375b18e6d8cda86 (127.25.14.1:36551)
I20260812 19:53:17.369294 27481 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:35869
I20260812 19:53:17.458933 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28409
I20260812 19:53:17.483788 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27813
I20260812 19:53:17.492446 27952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:17.580560 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27813
I20260812 19:53:17.609882 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27966
I20260812 19:53:17.618954 28101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:17.710693 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27966
I20260812 19:53:17.735735 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28117
I20260812 19:53:17.742575 28185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:17.851289 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28117
I20260812 19:53:17.881758 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27427
I20260812 19:53:17.883031 27514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:17.993678 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27427
I20260812 19:53:18.024812 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27522
I20260812 19:53:18.026228 27612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:18.137605 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27522
[ OK ] ExternalMiniClusterTest.TestBasicOperation/0 (14543 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/1
2026-08-12T19:53:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:18Z Disabled control of system clock
I20260812 19:53:18.212352 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
W20260812 19:53:18.580278 28577 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:18.580709 28577 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:18.580847 28577 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:18.589509 28577 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:18.589682 28577 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:18.589813 28577 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:18.589901 28577 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:18.600564 28577 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:18.602453 28577 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:18.604378 28577 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:18.613407 28583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:18.613363 28584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:18.613952 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:18.614952 28577 server_base.cc:1061] running on GCE node
I20260812 19:53:18.615816 28577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:18.617389 28577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:18.618690 28577 hybrid_clock.cc:648] HybridClock initialized: now 1786564398618609 us; error 77 us; skew 500 ppm
I20260812 19:53:18.619130 28577 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:18.621747 28577 webserver.cc:533] Webserver started at http://127.25.14.62:42407/ using document root <none> and password file <none>
I20260812 19:53:18.622648 28577 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:18.622807 28577 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:18.623231 28577 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:18.626044 28577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:18.627030 28577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:18.633422 28577 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:53:18.637405 28595 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:18.639124 28577 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:18.639353 28577 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:18.639626 28577 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:18.655725 28577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:18.656765 28577 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:18.657097 28577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:18.680089 28577 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35723
I20260812 19:53:18.680115 28652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35723 every 8 connection(s)
I20260812 19:53:18.681931 28577 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:18.686257 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28577
I20260812 19:53:18.686511 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:18.687047 28653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:18.687624 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41243
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
I20260812 19:53:18.693682 28653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:18.711476 28653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:18.713073 28599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
W20260812 19:53:18.715027 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
I20260812 19:53:18.764417 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } attempt: 1
W20260812 19:53:18.767097 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
I20260812 19:53:18.835420 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } attempt: 2
W20260812 19:53:18.838594 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
I20260812 19:53:18.929939 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } attempt: 3
W20260812 19:53:18.932341 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
I20260812 19:53:19.075690 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } attempt: 4
W20260812 19:53:19.078210 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111)
W20260812 19:53:19.101586 28655 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:19.102046 28655 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:19.102205 28655 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:19.111141 28655 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:19.111263 28655 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:19.111363 28655 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:19.111459 28655 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:19.122445 28655 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:19.124277 28655 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:19.126302 28655 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:19.135051 28664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:19.135020 28665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:19.135701 28667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:19.136270 28655 server_base.cc:1061] running on GCE node
I20260812 19:53:19.137189 28655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:19.138830 28655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:19.140084 28655 hybrid_clock.cc:648] HybridClock initialized: now 1786564399140035 us; error 48 us; skew 500 ppm
I20260812 19:53:19.140465 28655 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:19.142918 28655 webserver.cc:533] Webserver started at http://127.25.14.61:36297/ using document root <none> and password file <none>
I20260812 19:53:19.143774 28655 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:19.143934 28655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:19.144362 28655 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:19.147399 28655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.148352 28655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.154471 28655 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 19:53:19.158376 28675 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:19.159874 28655 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:19.160115 28655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.160386 28655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:19.177723 28655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:19.178696 28655 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:19.179018 28655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:19.201450 28655 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:41243
I20260812 19:53:19.201500 28731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:41243 every 8 connection(s)
I20260812 19:53:19.203280 28655 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:19.206489 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28655
I20260812 19:53:19.206676 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:19.207902 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35609
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
I20260812 19:53:19.208429 28732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:19.215270 28732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:19.230795 28732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:19.241448 28732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:19.242870 28678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
W20260812 19:53:19.244588 28732 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.293952 28732 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 1
W20260812 19:53:19.297766 28732 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.366065 28732 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 2
W20260812 19:53:19.370961 28732 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.378901 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } attempt: 5
I20260812 19:53:19.387573 28653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:19.390388 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.441684 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 1
W20260812 19:53:19.444533 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.462255 28732 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 3
W20260812 19:53:19.464962 28732 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.512862 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 2
W20260812 19:53:19.515592 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
W20260812 19:53:19.597355 28734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:19.597812 28734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:19.597929 28734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:19.606907 28734 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:19.607013 28734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:19.607105 28734 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:19.607195 28734 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:19.608242 28732 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 4
W20260812 19:53:19.610869 28732 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
I20260812 19:53:19.618886 28734 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:19.620738 28734 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:19.621866 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 3
I20260812 19:53:19.623215 28734 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:19.624466 28653 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111)
W20260812 19:53:19.632929 28749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:19.633193 28748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:19.634404 28751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:19.635569 28734 server_base.cc:1061] running on GCE node
I20260812 19:53:19.636533 28734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:19.638211 28734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:19.639515 28734 hybrid_clock.cc:648] HybridClock initialized: now 1786564399639455 us; error 45 us; skew 500 ppm
I20260812 19:53:19.639940 28734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:19.642597 28734 webserver.cc:533] Webserver started at http://127.25.14.60:38513/ using document root <none> and password file <none>
I20260812 19:53:19.643465 28734 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:19.643602 28734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:19.644018 28734 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:19.647104 28734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.648100 28734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.654409 28734 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 19:53:19.658474 28760 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:19.660180 28734 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:19.660418 28734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:19.660735 28734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:19.685920 28734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:19.687038 28734 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:19.687358 28734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:19.709388 28734 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:35609
I20260812 19:53:19.709414 28814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:35609 every 8 connection(s)
I20260812 19:53:19.711181 28734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:19.716336 28817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:19.717804 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28734
I20260812 19:53:19.718034 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:19.725056 28817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:19.741122 28817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:19.747943 28817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:19.759361 28817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Bootstrap starting.
I20260812 19:53:19.762992 28817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:19.764698 28817 log.cc:826] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:19.767812 28817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: No bootstrap required, opened a new log
I20260812 19:53:19.773586 28817 raft_consensus.cc:359] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.774192 28817 raft_consensus.cc:385] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:19.774370 28817 raft_consensus.cc:740] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b7e08ad9824c748fa847dbe0eb0f21, State: Initialized, Role: FOLLOWER
I20260812 19:53:19.775197 28817 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.776280 28823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.776602 28823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:19.777493 28817 sys_catalog.cc:565] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:53:19.790169 28835 catalog_manager.cc:1594] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:19.790352 28835 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:53:19.785857 28734 master_runner.cc:187] Error getting master registration for 127.25.14.62:35723: OK, instance_id { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" instance_seqno: 1786564398674036 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:19.791007 28734 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:19.801728 28653 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 4
I20260812 19:53:19.812345 28653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap starting.
I20260812 19:53:19.816545 28653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:19.817791 28653 log.cc:826] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:19.820451 28653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: No bootstrap required, opened a new log
I20260812 19:53:19.826545 28653 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.826722 28823 raft_consensus.cc:493] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:19.826911 28653 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:19.827072 28653 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de59a74a53c4c1089c1cc0d72e0c945, State: Initialized, Role: FOLLOWER
I20260812 19:53:19.827039 28823 raft_consensus.cc:515] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.827813 28653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.828949 28823 leader_election.cc:290] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723), 098e2bd1d13041a08e2edee9dddf9b0b (127.25.14.61:41243)
I20260812 19:53:19.828948 28838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.829262 28838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:19.830497 28653 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:19.830564 28624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" is_pre_election: true
I20260812 19:53:19.831112 28624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b7e08ad9824c748fa847dbe0eb0f21 in term 0.
I20260812 19:53:19.831939 28761 leader_election.cc:304] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de59a74a53c4c1089c1cc0d72e0c945, f9b7e08ad9824c748fa847dbe0eb0f21; no voters:
I20260812 19:53:19.832590 28823 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:53:19.832772 28823 raft_consensus.cc:493] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:19.832875 28823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:19.836611 28823 raft_consensus.cc:515] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.838424 28624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
I20260812 19:53:19.838748 28624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:19.840903 28823 leader_election.cc:290] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 1 election: Requested vote from peers 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723), 098e2bd1d13041a08e2edee9dddf9b0b (127.25.14.61:41243)
I20260812 19:53:19.842919 28624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b7e08ad9824c748fa847dbe0eb0f21 in term 1.
I20260812 19:53:19.843618 28761 leader_election.cc:304] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de59a74a53c4c1089c1cc0d72e0c945, f9b7e08ad9824c748fa847dbe0eb0f21; no voters:
I20260812 19:53:19.844076 28823 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:53:19.846717 28577 master_runner.cc:187] Error getting master registration for 127.25.14.61:41243: OK, instance_id { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" instance_seqno: 1786564399195376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:19.855927 28577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:19.856650 28849 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:19.856787 28849 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:53:19.857164 28823 raft_consensus.cc:697] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 LEADER]: Becoming Leader. State: Replica: f9b7e08ad9824c748fa847dbe0eb0f21, State: Running, Role: LEADER
I20260812 19:53:19.857838 28823 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.862102 28850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b7e08ad9824c748fa847dbe0eb0f21. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.862682 28850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:19.864368 28853 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:19.867311 28853 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:19.877295 28624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Refusing update from remote peer f9b7e08ad9824c748fa847dbe0eb0f21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:53:19.878387 28850 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:19.894454 28838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b7e08ad9824c748fa847dbe0eb0f21. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.894734 28838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:19.902117 28850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.902438 28850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:19.902971 28856 mvcc.cc:204] Tried to move back new op lower bound from 7317767781884215296 to 7317767781823918080. Current Snapshot: MvccSnapshot[applied={T|T < 7317767781884215296}]
I20260812 19:53:19.904230 28838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.904449 28838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:19.909219 28853 catalog_manager.cc:1383] Generated new cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:19.909332 28853 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:19.910081 28732 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } attempt: 5
I20260812 19:53:19.928241 28732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Bootstrap starting.
I20260812 19:53:19.936225 28732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:19.939069 28732 log.cc:826] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:19.970834 28732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: No bootstrap required, opened a new log
I20260812 19:53:19.973919 28853 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:53:19.982031 28853 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:19.984443 28732 raft_consensus.cc:359] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.984887 28732 raft_consensus.cc:385] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:19.985034 28732 raft_consensus.cc:740] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e2bd1d13041a08e2edee9dddf9b0b, State: Initialized, Role: FOLLOWER
I20260812 19:53:19.986043 28732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:19.993284 28861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:19.993711 28861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:19.995988 28732 sys_catalog.cc:565] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:19.996068 28702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
I20260812 19:53:19.996589 28702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:20.001330 28702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b7e08ad9824c748fa847dbe0eb0f21 in term 1.
I20260812 19:53:20.002691 28701 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Refusing update from remote peer f9b7e08ad9824c748fa847dbe0eb0f21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:53:19.996008 28703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" is_pre_election: true
I20260812 19:53:20.003789 28703 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f9b7e08ad9824c748fa847dbe0eb0f21 in term 1. Re-sending same reply.
I20260812 19:53:20.005508 28850 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:20.034858 28853 catalog_manager.cc:6092] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Generated new TSK 0
I20260812 19:53:20.036422 28853 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:20.044621 28861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b7e08ad9824c748fa847dbe0eb0f21. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:20.044934 28861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:20.045014 28823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:20.045310 28823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:20.045516 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:20.045864 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:20.055536 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:20.107084 28880 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 19:53:20.533397 28872 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:20.533780 28872 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:20.533962 28872 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:20.543049 28872 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:20.543308 28872 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:20.554422 28872 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:20.556298 28872 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:20.559161 28872 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:20.572897 28885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:20.573235 28886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:20.573743 28888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:20.574586 28872 server_base.cc:1061] running on GCE node
I20260812 19:53:20.575352 28872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:20.576541 28872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:20.577786 28872 hybrid_clock.cc:648] HybridClock initialized: now 1786564400577711 us; error 64 us; skew 500 ppm
I20260812 19:53:20.578141 28872 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:20.580678 28872 webserver.cc:533] Webserver started at http://127.25.14.1:38233/ using document root <none> and password file <none>
I20260812 19:53:20.581733 28872 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:20.581890 28872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:20.582306 28872 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:20.585008 28872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:20.585932 28872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:20.591969 28872 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:53:20.596246 28896 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:20.597990 28872 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:20.598205 28872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:20.598470 28872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:20.627293 28872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:20.628377 28872 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:20.628729 28872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:20.629973 28872 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:20.632002 28872 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:20.632124 28872 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:20.632274 28872 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:20.632382 28872 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:20.672725 28872 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36919
I20260812 19:53:20.672788 29016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36919 every 8 connection(s)
I20260812 19:53:20.674402 28872 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:20.675844 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28872
I20260812 19:53:20.676050 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:20.684031 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:20.720582 29018 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:20.721065 29018 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:20.722417 29018 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:20.724717 29017 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:20.724915 29017 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:20.725528 29017 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:20.726615 28611 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
I20260812 19:53:20.727278 29019 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:20.727423 29019 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:20.727533 28777 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
I20260812 19:53:20.728018 29019 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:20.729768 28692 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
I20260812 19:53:20.729728 28777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:46117
I20260812 19:53:20.858970 28849 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:20.859117 28849 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: loading cluster ID for follower catalog manager: success
I20260812 19:53:20.861923 28849 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: acquiring CA information for follower catalog manager: success
I20260812 19:53:20.863907 28849 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:53:21.087626 29028 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:21.087996 29028 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:21.088164 29028 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:21.096869 29028 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:21.097097 29028 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:21.107832 29028 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:21.109710 29028 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:21.111644 29028 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:21.112710 28880 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:21.112857 28880 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: loading cluster ID for follower catalog manager: success
I20260812 19:53:21.116940 28880 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: acquiring CA information for follower catalog manager: success
I20260812 19:53:21.119585 28880 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:53:21.122294 29036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:21.122905 29035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:21.124606 29038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:21.125404 29028 server_base.cc:1061] running on GCE node
I20260812 19:53:21.126094 29028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:21.127148 29028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:21.128414 29028 hybrid_clock.cc:648] HybridClock initialized: now 1786564401128347 us; error 59 us; skew 500 ppm
I20260812 19:53:21.128839 29028 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:21.131521 29028 webserver.cc:533] Webserver started at http://127.25.14.2:35221/ using document root <none> and password file <none>
I20260812 19:53:21.132386 29028 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:21.132544 29028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:21.132970 29028 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:21.135852 29028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "b89a527fca624c3e8a53b5309ada479c"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.136813 29028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "b89a527fca624c3e8a53b5309ada479c"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.144214 29028 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 19:53:21.148569 29045 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.150439 29028 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:53:21.150660 29028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "b89a527fca624c3e8a53b5309ada479c"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.150921 29028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:21.178041 29028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:21.179124 29028 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:21.179446 29028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:21.180578 29028 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:21.182660 29028 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:21.182796 29028 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.182927 29028 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:21.183019 29028 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.232120 29028 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:35391
I20260812 19:53:21.232230 29166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:35391 every 8 connection(s)
I20260812 19:53:21.233870 29028 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:21.241286 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29028
I20260812 19:53:21.241531 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:21.287743 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:21.298695 29168 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:21.299206 29168 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.299767 29167 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:21.299919 29167 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.300554 29167 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:21.300622 29168 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:21.302608 28777 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:35391)
I20260812 19:53:21.302644 28611 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:35391)
I20260812 19:53:21.303798 28777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:56011
I20260812 19:53:21.309990 29170 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:21.310213 29170 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.310741 29170 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:21.312208 28692 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:35391)
W20260812 19:53:21.672232 29180 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:21.672730 29180 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:21.672864 29180 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:21.684954 29180 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:21.685222 29180 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:21.696097 29180 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:21.698312 29180 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:21.700562 29180 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:21.712226 29188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:21.712786 29186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:21.713547 29190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:21.720160 29180 server_base.cc:1061] running on GCE node
I20260812 19:53:21.721010 29180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:21.722242 29180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:21.723505 29180 hybrid_clock.cc:648] HybridClock initialized: now 1786564401723448 us; error 49 us; skew 500 ppm
I20260812 19:53:21.723872 29180 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:21.726460 29180 webserver.cc:533] Webserver started at http://127.25.14.3:46807/ using document root <none> and password file <none>
I20260812 19:53:21.727373 29180 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:21.727548 29180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:21.727977 29180 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:21.731526 29180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "e8b3e85fd74f4f138346e85202b7d00d"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.732613 29180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "e8b3e85fd74f4f138346e85202b7d00d"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.733021 29017 heartbeater.cc:499] Master 127.25.14.60:35609 was elected leader, sending a full tablet report...
I20260812 19:53:21.739583 29180 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260812 19:53:21.743665 29197 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.745398 29180 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 19:53:21.745675 29180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "e8b3e85fd74f4f138346e85202b7d00d"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:21.745967 29180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:21.766006 29180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:21.767028 29180 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:21.767371 29180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:21.768378 29180 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:21.770378 29180 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:21.770485 29180 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.770565 29180 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:21.770624 29180 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:21.811807 29180 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:40209
I20260812 19:53:21.811865 29319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:40209 every 8 connection(s)
I20260812 19:53:21.813438 29180 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:21.821321 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29180
I20260812 19:53:21.821563 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:21.853716 29322 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:21.853997 29323 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:21.854149 29322 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.854203 29323 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.854487 29321 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:21.854754 29321 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:21.855535 29321 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:21.855561 29322 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:21.856426 29323 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:21.857446 28692 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:40209)
I20260812 19:53:21.857448 28611 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:40209)
I20260812 19:53:21.857511 28777 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:40209)
I20260812 19:53:21.858485 28777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:39795
I20260812 19:53:21.871382 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:21.879777 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:21.879987 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28577
I20260812 19:53:21.881045 28647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:21.991127 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28577
I20260812 19:53:22.021442 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--webserver_interface=127.25.14.62
--webserver_port=42407
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
W20260812 19:53:22.158668 28761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35723: connect: Connection refused (error 111)
W20260812 19:53:22.163215 28761 consensus_peers.cc:597] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 -> Peer 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723): Couldn't send request to peer 6de59a74a53c4c1089c1cc0d72e0c945. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:53:22.306152 29168 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35723: connect: Connection refused (error 111)
I20260812 19:53:22.307089 29167 heartbeater.cc:499] Master 127.25.14.60:35609 was elected leader, sending a full tablet report...
W20260812 19:53:22.403863 29333 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:22.404240 29333 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:22.404317 29333 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:22.413446 29333 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:22.413570 29333 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:22.413705 29333 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:22.413782 29333 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:22.424520 29333 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=42407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:22.426451 29333 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:22.428431 29333 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:22.437575 29342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:22.437721 29345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:22.438473 29343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:22.439419 29333 server_base.cc:1061] running on GCE node
I20260812 19:53:22.440341 29333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:22.442047 29333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:22.443305 29333 hybrid_clock.cc:648] HybridClock initialized: now 1786564402443256 us; error 68 us; skew 500 ppm
I20260812 19:53:22.443706 29333 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:22.446333 29333 webserver.cc:533] Webserver started at http://127.25.14.62:42407/ using document root <none> and password file <none>
I20260812 19:53:22.447268 29333 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:22.447463 29333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:22.454005 29333 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:22.457319 29351 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:22.458889 29333 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:22.459108 29333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:22.459966 29333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:22.479888 29333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:22.480863 29333 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:22.481227 29333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:22.502929 29333 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35723
I20260812 19:53:22.502950 29407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35723 every 8 connection(s)
I20260812 19:53:22.504931 29333 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:22.507383 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29333
I20260812 19:53:22.507753 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28872
I20260812 19:53:22.514314 29408 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:22.518249 29012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:22.519129 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap starting.
I20260812 19:53:22.544572 29408 log.cc:826] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:22.556005 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:22.556716 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap complete.
I20260812 19:53:22.567657 29408 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:22.568717 29408 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de59a74a53c4c1089c1cc0d72e0c945, State: Initialized, Role: FOLLOWER
I20260812 19:53:22.569695 29408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:22.570617 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:22.570920 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:22.572011 29408 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:22.584293 29426 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:22.584543 29426 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: loading cluster ID for follower catalog manager: success
I20260812 19:53:22.589190 29426 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: acquiring CA information for follower catalog manager: success
I20260812 19:53:22.592249 29426 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:22.634735 29434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:22.635102 29434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:22.640184 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b7e08ad9824c748fa847dbe0eb0f21. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:22.640524 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:22.651461 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28872
I20260812 19:53:22.680896 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36919
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=38233
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:22.860793 29321 heartbeater.cc:499] Master 127.25.14.60:35609 was elected leader, sending a full tablet report...
I20260812 19:53:22.871884 29369 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e8b3e85fd74f4f138346e85202b7d00d" instance_seqno: 1786564401800400) as {username='slave'} at 127.25.14.3:60415; Asking this server to re-register.
I20260812 19:53:22.872870 29322 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:22.874888 29369 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:40209)
I20260812 19:53:23.023705 29434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:23.024010 29434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: LEADER
W20260812 19:53:23.069533 29436 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:23.069989 29436 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:23.070081 29436 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:23.079264 29436 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:23.079447 29436 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:23.089900 29436 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36919
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38233
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:23.091624 29436 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:23.093544 29436 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:23.104079 29442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:23.104209 29445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:23.104101 29443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:23.105422 29436 server_base.cc:1061] running on GCE node
I20260812 19:53:23.106189 29436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:23.107249 29436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:23.108505 29436 hybrid_clock.cc:648] HybridClock initialized: now 1786564403108437 us; error 60 us; skew 500 ppm
I20260812 19:53:23.108865 29436 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:23.111346 29436 webserver.cc:533] Webserver started at http://127.25.14.1:38233/ using document root <none> and password file <none>
I20260812 19:53:23.112234 29436 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:23.112428 29436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:23.118745 29436 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:53:23.121915 29452 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:23.123400 29436 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:53:23.123600 29436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:23.124374 29436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:23.150665 29436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:23.151774 29436 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:23.152125 29436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:23.153235 29436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:23.155176 29436 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:23.155287 29436 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:23.155449 29436 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:23.155526 29436 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:23.191931 29436 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36919
I20260812 19:53:23.191975 29579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36919 every 8 connection(s)
I20260812 19:53:23.193681 29436 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:23.198344 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29436
I20260812 19:53:23.198700 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29436
I20260812 19:53:23.223965 29581 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:23.224464 29581 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:23.225142 29582 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:23.225314 29582 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:23.225852 29581 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:23.225986 29582 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:23.227129 29572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:23.228063 28692 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
I20260812 19:53:23.228106 29369 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
I20260812 19:53:23.314045 29168 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:23.315188 29369 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b89a527fca624c3e8a53b5309ada479c" instance_seqno: 1786564401222174) as {username='slave'} at 127.25.14.2:41923; Asking this server to re-register.
I20260812 19:53:23.315841 29168 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:23.317096 29369 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:35391)
I20260812 19:53:23.319507 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29436
I20260812 19:53:23.345706 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36919
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=38233
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260812 19:53:23.345983 29584 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:53:23.197360 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:57867 (local address 127.25.14.60:35609)
0812 19:53:23.222036 (+ 24676us) server_negotiation.cc:207] Beginning negotiation
0812 19:53:23.222041 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:53:23.222061 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:53:23.222116 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:53:23.222121 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:53:23.222243 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:53:23.223416 (+ 1173us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:53:23.223598 (+ 182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:53:23.229596 (+ 5998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:53:23.344956 (+115360us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:57867: BlockingRecv error: recv got EOF from 127.25.14.1:57867 (error 108)
Metrics: {"server-negotiator.queue_time_us":24555,"thread_start_us":24420,"threads_started":1}
W20260812 19:53:23.723483 29593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:23.723920 29593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:23.724071 29593 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:23.724171 29593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:23.732971 29593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:23.733266 29593 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:23.744150 29593 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36919
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38233
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:23.746161 29593 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:23.748292 29593 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:23.760017 29599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:23.760512 29601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:23.760007 29598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:23.761296 29593 server_base.cc:1061] running on GCE node
I20260812 19:53:23.762172 29593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:23.763223 29593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:23.764490 29593 hybrid_clock.cc:648] HybridClock initialized: now 1786564403764387 us; error 49 us; skew 500 ppm
I20260812 19:53:23.764838 29593 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:23.767735 29593 webserver.cc:533] Webserver started at http://127.25.14.1:38233/ using document root <none> and password file <none>
I20260812 19:53:23.768599 29593 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:23.768764 29593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:23.775736 29593 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 19:53:23.779282 29608 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:23.781203 29593 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:53:23.781416 29593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:23.782366 29593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:23.798810 29593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:23.799883 29593 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:23.800249 29593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:23.801376 29593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:23.803340 29593 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:23.803438 29593 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:23.803611 29593 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:23.803695 29593 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:23.804105 29593 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:23.804241 29593 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:23.804325 29593 fault_injection.cc:59] injecting fault for kudu (pid 29593): FLAGS_fault_before_start (process will exit)
I20260812 19:53:23.831439 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36919
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=38233
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:53:24.202183 29616 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:24.202620 29616 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:24.202778 29616 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:24.211617 29616 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:24.211864 29616 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:24.222576 29616 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36919
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38233
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:24.224500 29616 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:24.226549 29616 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:24.237502 29622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:24.237471 29624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:24.238824 29626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:24.239569 29616 server_base.cc:1061] running on GCE node
I20260812 19:53:24.240439 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:24.241478 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:24.242816 29616 hybrid_clock.cc:648] HybridClock initialized: now 1786564404242726 us; error 77 us; skew 500 ppm
I20260812 19:53:24.243180 29616 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:24.245426 29616 webserver.cc:533] Webserver started at http://127.25.14.1:38233/ using document root <none> and password file <none>
I20260812 19:53:24.246335 29616 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:24.246498 29616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:24.252357 29616 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:53:24.255278 29633 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:24.256884 29616 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 19:53:24.257069 29616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:24.257938 29616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:24.277746 29616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:24.278774 29616 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:24.279143 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:24.280293 29616 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:24.282330 29616 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:24.282460 29616 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:24.282634 29616 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:24.282709 29616 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:24.334700 29616 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36919
I20260812 19:53:24.334766 29756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36919 every 8 connection(s)
I20260812 19:53:24.336391 29616 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:24.337865 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29616
I20260812 19:53:24.338209 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29616
I20260812 19:53:24.372169 29752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:24.461222 29759 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:24.461751 29760 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:24.462015 29760 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:24.463333 29760 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:24.464356 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29616
I20260812 19:53:24.467029 28692 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:36919)
W20260812 19:53:24.489528 28678 connection.cc:573] server connection from 127.25.14.1:57103 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:24.489871 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29028
I20260812 19:53:24.498428 29162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:24.590206 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29028
I20260812 19:53:24.615481 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29180
I20260812 19:53:24.624539 29315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:24.713238 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29180
I20260812 19:53:24.739329 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29333
I20260812 19:53:24.746412 29401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:24.856680 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29333
W20260812 19:53:24.887349 28761 connection.cc:573] client connection to 127.25.14.62:35723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:24.887600 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28655
I20260812 19:53:24.887725 28734 master_runner.cc:217] Error getting master consensus for 127.25.14.62:35723: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:24.888037 28734 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:24.888913 28727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:24.997124 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28655
I20260812 19:53:25.026764 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28734
I20260812 19:53:25.028095 28809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:25.140323 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28734
2026-08-12T19:53:25Z chronyd exiting
2026-08-12T19:53:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:25Z Disabled control of system clock
I20260812 19:53:25.221292 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
W20260812 19:53:25.587298 29783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:25.587735 29783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:25.587886 29783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:25.596665 29783 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:25.596781 29783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:25.596889 29783 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:25.596983 29783 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:25.607654 29783 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:25.609488 29783 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:25.611488 29783 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:25.620556 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:25.620566 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:25.621413 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:25.622069 29783 server_base.cc:1061] running on GCE node
I20260812 19:53:25.622947 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:25.624498 29783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:25.625744 29783 hybrid_clock.cc:648] HybridClock initialized: now 1786564405625687 us; error 37 us; skew 500 ppm
I20260812 19:53:25.626194 29783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:25.628660 29783 webserver.cc:533] Webserver started at http://127.25.14.62:41587/ using document root <none> and password file <none>
I20260812 19:53:25.629525 29783 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:25.629690 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:25.635799 29783 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 19:53:25.639289 29798 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:25.640934 29783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:25.641161 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:25.642084 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:25.679055 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:25.680063 29783 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:25.680388 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:25.703698 29783 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35723
I20260812 19:53:25.703729 29857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35723 every 8 connection(s)
I20260812 19:53:25.705581 29783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:25.707304 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29783
I20260812 19:53:25.707508 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:25.708807 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41243
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
I20260812 19:53:25.716253 29858 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:25.720218 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap starting.
I20260812 19:53:25.738106 29858 log.cc:826] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:25.748524 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:25.749151 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap complete.
I20260812 19:53:25.756528 29858 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:25.757309 29858 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de59a74a53c4c1089c1cc0d72e0c945, State: Initialized, Role: FOLLOWER
I20260812 19:53:25.758301 29858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:25.759351 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:25.759735 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:25.760421 29858 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:25.772691 29873 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:25.772953 29873 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: loading cluster ID for follower catalog manager: success
I20260812 19:53:25.777945 29873 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: acquiring CA information for follower catalog manager: success
I20260812 19:53:25.781317 29873 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:25.795293 29783 master_runner.cc:187] Error getting master registration for 127.25.14.61:41243: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41243: connect: Connection refused (error 111),
I20260812 19:53:25.801241 29783 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:26.107331 29860 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:26.107710 29860 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:26.107846 29860 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:26.116887 29860 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:26.117004 29860 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:26.117064 29860 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:26.117187 29860 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:26.128244 29860 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:26.130241 29860 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:26.132279 29860 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:26.141242 29882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:26.141451 29881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:26.142671 29884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:26.143945 29860 server_base.cc:1061] running on GCE node
I20260812 19:53:26.144963 29860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:26.146682 29860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:26.148000 29860 hybrid_clock.cc:648] HybridClock initialized: now 1786564406147893 us; error 106 us; skew 500 ppm
I20260812 19:53:26.148571 29860 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:26.151396 29860 webserver.cc:533] Webserver started at http://127.25.14.61:43207/ using document root <none> and password file <none>
I20260812 19:53:26.152364 29860 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:26.152547 29860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:26.159552 29860 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 19:53:26.163022 29890 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:26.164719 29860 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:53:26.164971 29860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:26.165961 29860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:26.199209 29860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:26.200300 29860 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:26.200663 29860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:26.223526 29860 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:41243
I20260812 19:53:26.223559 29946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:41243 every 8 connection(s)
I20260812 19:53:26.225447 29860 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:26.226740 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29860
I20260812 19:53:26.226979 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:26.228289 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35609
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
I20260812 19:53:26.236651 29947 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:26.240828 29947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Bootstrap starting.
I20260812 19:53:26.259446 29947 log.cc:826] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:26.268999 29947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:26.269593 29947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Bootstrap complete.
I20260812 19:53:26.277076 29947 raft_consensus.cc:359] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:26.277961 29947 raft_consensus.cc:740] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e2bd1d13041a08e2edee9dddf9b0b, State: Initialized, Role: FOLLOWER
I20260812 19:53:26.278769 29947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:26.279798 29951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:26.280131 29951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:26.280993 29947 sys_catalog.cc:565] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:26.293978 29964 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:26.294261 29964 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: loading cluster ID for follower catalog manager: success
I20260812 19:53:26.300613 29964 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: acquiring CA information for follower catalog manager: success
I20260812 19:53:26.303432 29964 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:26.335170 29860 master_runner.cc:187] Error getting master registration for 127.25.14.60:35609: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35609: connect: Connection refused (error 111),
I20260812 19:53:26.340895 29860 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:26.628173 29949 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:26.628545 29949 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:26.628667 29949 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:26.637508 29949 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:26.637676 29949 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:26.637804 29949 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:26.637859 29949 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:26.648720 29949 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:26.650605 29949 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:26.652556 29949 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:26.661556 29976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:26.662963 29975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:26.663817 29978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:26.665140 29949 server_base.cc:1061] running on GCE node
I20260812 19:53:26.666193 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:26.667944 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:26.669186 29949 hybrid_clock.cc:648] HybridClock initialized: now 1786564406669114 us; error 78 us; skew 500 ppm
I20260812 19:53:26.669584 29949 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:26.672397 29949 webserver.cc:533] Webserver started at http://127.25.14.60:39253/ using document root <none> and password file <none>
I20260812 19:53:26.673273 29949 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:26.673418 29949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:26.679862 29949 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:53:26.683732 29984 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:26.685710 29949 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:53:26.685920 29949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
format_stamp: "Formatted at 2026-08-12 19:53:19 on dist-test-slave-p3t5"
I20260812 19:53:26.686708 29949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:26.714514 29949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:26.715512 29949 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:26.715830 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:26.738148 29949 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:35609
I20260812 19:53:26.738173 30039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:35609 every 8 connection(s)
I20260812 19:53:26.740029 29949 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:26.746719 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29949
I20260812 19:53:26.746942 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:26.748719 30040 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:26.753923 30040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Bootstrap starting.
I20260812 19:53:26.776165 30040 log.cc:826] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:26.787107 30040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:26.787744 30040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: Bootstrap complete.
I20260812 19:53:26.794901 30040 raft_consensus.cc:359] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:26.795676 30040 raft_consensus.cc:740] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b7e08ad9824c748fa847dbe0eb0f21, State: Initialized, Role: FOLLOWER
I20260812 19:53:26.796370 30040 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:26.797174 30045 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:26.797435 30045 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:26.798398 30040 sys_catalog.cc:565] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:26.810923 30058 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:26.811149 30058 catalog_manager.cc:1588] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: loading cluster ID for follower catalog manager: success
I20260812 19:53:26.816236 30058 catalog_manager.cc:1610] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: acquiring CA information for follower catalog manager: success
I20260812 19:53:26.819510 30058 catalog_manager.cc:1638] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:26.840131 29949 master_runner.cc:264] No leader master found from master f9b7e08ad9824c748fa847dbe0eb0f21
I20260812 19:53:26.840281 29949 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:27.489761 30061 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:27.490140 30061 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:27.492578 30061 leader_election.cc:290] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 098e2bd1d13041a08e2edee9dddf9b0b (127.25.14.61:41243), f9b7e08ad9824c748fa847dbe0eb0f21 (127.25.14.60:35609)
I20260812 19:53:27.502418 30014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" is_pre_election: true
I20260812 19:53:27.502429 29920 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" is_pre_election: true
I20260812 19:53:27.503006 30014 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de59a74a53c4c1089c1cc0d72e0c945 in term 1.
I20260812 19:53:27.503113 29920 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de59a74a53c4c1089c1cc0d72e0c945 in term 1.
I20260812 19:53:27.503757 29801 leader_election.cc:304] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de59a74a53c4c1089c1cc0d72e0c945, f9b7e08ad9824c748fa847dbe0eb0f21; no voters:
I20260812 19:53:27.504202 30061 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:53:27.504375 30061 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:27.504492 30061 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:27.508345 30061 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:27.509059 30061 leader_election.cc:290] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [CANDIDATE]: Term 2 election: Requested vote from peers 098e2bd1d13041a08e2edee9dddf9b0b (127.25.14.61:41243), f9b7e08ad9824c748fa847dbe0eb0f21 (127.25.14.60:35609)
I20260812 19:53:27.509727 30014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
I20260812 19:53:27.509693 29920 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "098e2bd1d13041a08e2edee9dddf9b0b"
I20260812 19:53:27.509981 30014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:27.509981 29920 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:27.516220 29920 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de59a74a53c4c1089c1cc0d72e0c945 in term 2.
I20260812 19:53:27.516220 30014 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de59a74a53c4c1089c1cc0d72e0c945 in term 2.
I20260812 19:53:27.516836 29801 leader_election.cc:304] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de59a74a53c4c1089c1cc0d72e0c945, f9b7e08ad9824c748fa847dbe0eb0f21; no voters:
I20260812 19:53:27.517221 30061 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:53:27.517733 30061 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 LEADER]: Becoming Leader. State: Replica: 6de59a74a53c4c1089c1cc0d72e0c945, State: Running, Role: LEADER
I20260812 19:53:27.518319 30061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:27.521417 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de59a74a53c4c1089c1cc0d72e0c945. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.521800 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:27.522531 30068 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:27.524739 30068 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:27.525729 30068 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:27.525818 30068 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:27.527014 30068 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:27.527871 30068 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Loaded TSK: 0
I20260812 19:53:27.528864 30068 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:27.535543 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:53:27.919916 30069 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:27.920284 30069 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:27.920415 30069 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:27.929488 30069 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:27.929710 30069 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:27.933745 29920 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Refusing update from remote peer 6de59a74a53c4c1089c1cc0d72e0c945: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:53:27.935075 30066 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:27.939664 30074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de59a74a53c4c1089c1cc0d72e0c945. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.939975 30074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:27.941326 30061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.941735 30061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:27.943819 30014 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 2 FOLLOWER]: Refusing update from remote peer 6de59a74a53c4c1089c1cc0d72e0c945: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:53:27.944248 30069 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:27.945313 30061 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:27.947108 30069 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:27.950656 30069 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:27.953178 30074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.953429 30074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:27.959244 30083 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.959621 30083 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:27.960067 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de59a74a53c4c1089c1cc0d72e0c945. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:27.960433 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:27.964960 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:27.965121 30086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:27.965673 30088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:27.966701 30069 server_base.cc:1061] running on GCE node
I20260812 19:53:27.967422 30069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:27.968487 30069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:27.969722 30069 hybrid_clock.cc:648] HybridClock initialized: now 1786564407969668 us; error 46 us; skew 500 ppm
I20260812 19:53:27.970119 30069 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:27.972762 30069 webserver.cc:533] Webserver started at http://127.25.14.1:39721/ using document root <none> and password file <none>
I20260812 19:53:27.973695 30069 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:27.973855 30069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:27.979986 30069 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 19:53:27.983577 30094 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:27.985306 30069 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:53:27.985508 30069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:27.986522 30069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:28.012805 30069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:28.013898 30069 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:28.014226 30069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:28.015281 30069 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:28.017238 30069 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:28.017354 30069 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:28.017514 30069 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:28.017593 30069 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:28.055411 30069 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40247
I20260812 19:53:28.055478 30224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40247 every 8 connection(s)
I20260812 19:53:28.057202 30069 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:28.064275 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30069
I20260812 19:53:28.064570 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:28.072504 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:28.112632 30225 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:28.112689 30227 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:28.112824 30226 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:28.113085 30226 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.113080 30225 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.113080 30227 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.114621 30225 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:28.114658 30227 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:28.114719 30226 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:28.117455 30004 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:28.117468 29910 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:28.117460 29819 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:28.120056 29819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41339
I20260812 19:53:28.426849 30061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:28.427129 30061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: LEADER
W20260812 19:53:28.470988 30235 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:28.471335 30235 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:28.471460 30235 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:28.480273 30235 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:28.480455 30235 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:28.491302 30235 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:28.493182 30235 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:28.495074 30235 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:28.505541 30240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:28.505558 30241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:28.506285 30243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:28.507059 30235 server_base.cc:1061] running on GCE node
I20260812 19:53:28.507689 30235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:28.508692 30235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:28.509928 30235 hybrid_clock.cc:648] HybridClock initialized: now 1786564408509872 us; error 58 us; skew 500 ppm
I20260812 19:53:28.510277 30235 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:28.512609 30235 webserver.cc:533] Webserver started at http://127.25.14.2:34467/ using document root <none> and password file <none>
I20260812 19:53:28.513470 30235 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:28.513693 30235 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:28.519721 30235 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 19:53:28.523000 30250 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:28.524500 30235 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:53:28.524709 30235 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "b89a527fca624c3e8a53b5309ada479c"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:28.525521 30235 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:28.547717 30235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:28.548736 30235 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:28.549073 30235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:28.550233 30235 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:28.552117 30235 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:28.552230 30235 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:28.552397 30235 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:28.552471 30235 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:28.598682 30235 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:42125
I20260812 19:53:28.598779 30372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:42125 every 8 connection(s)
I20260812 19:53:28.600428 30235 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:28.608908 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30235
I20260812 19:53:28.609125 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:28.614017 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:28.623642 30374 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:28.624207 30374 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.625741 30374 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:28.627796 29819 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:42125)
I20260812 19:53:28.628971 29819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:55839
I20260812 19:53:28.632400 30373 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:28.632568 30373 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.633090 30373 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:28.633425 30375 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:28.633585 30375 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:28.633955 30375 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:28.634486 30004 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:42125)
I20260812 19:53:28.635147 29910 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:42125)
W20260812 19:53:29.021869 30381 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:29.022315 30381 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:29.022470 30381 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:29.031522 30381 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:29.031764 30381 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:29.042506 30381 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:29.044369 30381 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:29.046593 30381 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:29.057057 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:29.057036 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:29.058766 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:29.059417 30381 server_base.cc:1061] running on GCE node
I20260812 19:53:29.060045 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:29.061035 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:29.062282 30381 hybrid_clock.cc:648] HybridClock initialized: now 1786564409062213 us; error 60 us; skew 500 ppm
I20260812 19:53:29.062638 30381 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:29.065151 30381 webserver.cc:533] Webserver started at http://127.25.14.3:44283/ using document root <none> and password file <none>
I20260812 19:53:29.066474 30381 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:29.066668 30381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:29.073191 30381 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:53:29.076326 30396 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:29.078055 30381 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:29.078274 30381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "e8b3e85fd74f4f138346e85202b7d00d"
format_stamp: "Formatted at 2026-08-12 19:53:21 on dist-test-slave-p3t5"
I20260812 19:53:29.079097 30381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:29.093600 30381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:29.094640 30381 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:29.094954 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:29.096009 30381 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:29.097911 30381 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:29.097988 30381 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:29.098168 30381 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:29.098222 30381 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:29.123108 30226 heartbeater.cc:499] Master 127.25.14.62:35723 was elected leader, sending a full tablet report...
I20260812 19:53:29.137975 30381 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:41849
I20260812 19:53:29.138022 30524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:41849 every 8 connection(s)
I20260812 19:53:29.139726 30381 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:29.148833 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30381
I20260812 19:53:29.149144 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:29.183205 30527 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:29.183225 30526 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:29.183322 30525 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:29.183691 30525 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:29.183694 30527 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:29.183697 30526 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:29.185261 30527 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:29.185153 30525 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:29.185294 30526 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:29.187281 29819 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:41849)
I20260812 19:53:29.187361 30004 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:41849)
I20260812 19:53:29.187460 29910 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:41849)
I20260812 19:53:29.188349 29819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:38183
I20260812 19:53:29.197207 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:29.205124 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:29.205339 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29783
I20260812 19:53:29.206395 29853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:29.327252 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29783
I20260812 19:53:29.357246 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--webserver_interface=127.25.14.62
--webserver_port=41587
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609 with env {}
I20260812 19:53:29.633193 30374 heartbeater.cc:499] Master 127.25.14.62:35723 was elected leader, sending a full tablet report...
W20260812 19:53:29.635150 30374 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35723: connect: Connection refused (error 111)
W20260812 19:53:29.733229 30538 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:29.733631 30538 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:29.733812 30538 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:29.742915 30538 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:29.743072 30538 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:29.743186 30538 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:29.743273 30538 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:29.754341 30538 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=41587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:29.756174 30538 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:29.758250 30538 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:29.767619 30544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:29.767791 30547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:29.768350 30545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:29.769026 30538 server_base.cc:1061] running on GCE node
I20260812 19:53:29.769997 30538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:29.771720 30538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:29.772979 30538 hybrid_clock.cc:648] HybridClock initialized: now 1786564409772899 us; error 65 us; skew 500 ppm
I20260812 19:53:29.773408 30538 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:29.776000 30538 webserver.cc:533] Webserver started at http://127.25.14.62:41587/ using document root <none> and password file <none>
I20260812 19:53:29.776906 30538 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:29.777040 30538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:29.783864 30538 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:53:29.787887 30553 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:29.789537 30538 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:53:29.789798 30538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
format_stamp: "Formatted at 2026-08-12 19:53:18 on dist-test-slave-p3t5"
I20260812 19:53:29.790731 30538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:29.816036 30538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:29.817056 30538 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:29.817381 30538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:29.840453 30538 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35723
I20260812 19:53:29.840466 30610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35723 every 8 connection(s)
I20260812 19:53:29.842914 30538 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:29.851502 30611 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:29.853101 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30538
I20260812 19:53:29.853433 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30069
I20260812 19:53:29.856146 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap starting.
I20260812 19:53:29.863169 30220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:29.882608 30611 log.cc:826] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:29.896499 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:29.897294 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: Bootstrap complete.
I20260812 19:53:29.909098 30611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:29.910276 30611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de59a74a53c4c1089c1cc0d72e0c945, State: Initialized, Role: FOLLOWER
I20260812 19:53:29.911269 30611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:29.912223 30618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:29.912566 30618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:29.913837 30611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:29.926013 30630 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:29.926257 30630 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: loading cluster ID for follower catalog manager: success
I20260812 19:53:29.930966 30630 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: acquiring CA information for follower catalog manager: success
I20260812 19:53:29.933959 30630 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:29.977663 30538 master_runner.cc:264] No leader master found from master 6de59a74a53c4c1089c1cc0d72e0c945
I20260812 19:53:29.977864 30538 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:29.989566 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30069
I20260812 19:53:30.017364 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40247
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=39721
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:30.190718 30526 heartbeater.cc:499] Master 127.25.14.62:35723 was elected leader, sending a full tablet report...
I20260812 19:53:30.202452 30571 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e8b3e85fd74f4f138346e85202b7d00d" instance_seqno: 1786564409128872) as {username='slave'} at 127.25.14.3:51941; Asking this server to re-register.
I20260812 19:53:30.203619 30526 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:30.205888 30571 ts_manager.cc:194] Registered new tserver with Master: e8b3e85fd74f4f138346e85202b7d00d (127.25.14.3:41849)
W20260812 19:53:30.400786 30636 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:30.401175 30636 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:30.401312 30636 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:30.410362 30636 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:30.410598 30636 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:30.420416 30642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6de59a74a53c4c1089c1cc0d72e0c945)
I20260812 19:53:30.420715 30642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:30.421936 30636 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40247
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39721
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:30.423019 30642 leader_election.cc:290] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723), f9b7e08ad9824c748fa847dbe0eb0f21 (127.25.14.60:35609)
I20260812 19:53:30.424207 30636 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:30.427013 30636 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:30.431267 30014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" is_pre_election: true
I20260812 19:53:30.432333 30583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" is_pre_election: true
I20260812 19:53:30.432906 30583 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e2bd1d13041a08e2edee9dddf9b0b in term 2.
I20260812 19:53:30.433651 29891 leader_election.cc:304] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 098e2bd1d13041a08e2edee9dddf9b0b, 6de59a74a53c4c1089c1cc0d72e0c945; no voters: f9b7e08ad9824c748fa847dbe0eb0f21
I20260812 19:53:30.434121 30642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:53:30.434279 30642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6de59a74a53c4c1089c1cc0d72e0c945)
I20260812 19:53:30.434429 30642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:53:30.436916 30642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:30.437975 30642 leader_election.cc:290] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [CANDIDATE]: Term 3 election: Requested vote from peers 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723), f9b7e08ad9824c748fa847dbe0eb0f21 (127.25.14.60:35609)
I20260812 19:53:30.438953 30014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21"
I20260812 19:53:30.439247 30583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6de59a74a53c4c1089c1cc0d72e0c945"
I20260812 19:53:30.439546 30583 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:53:30.441311 30647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:30.442864 30649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:30.443034 30646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:30.443408 30636 server_base.cc:1061] running on GCE node
I20260812 19:53:30.444168 30636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:30.445032 30583 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e2bd1d13041a08e2edee9dddf9b0b in term 3.
I20260812 19:53:30.445328 30636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:30.445722 29891 leader_election.cc:304] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 098e2bd1d13041a08e2edee9dddf9b0b, 6de59a74a53c4c1089c1cc0d72e0c945; no voters: f9b7e08ad9824c748fa847dbe0eb0f21
I20260812 19:53:30.446105 30642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:53:30.446607 30636 hybrid_clock.cc:648] HybridClock initialized: now 1786564410446515 us; error 70 us; skew 500 ppm
I20260812 19:53:30.446583 30642 raft_consensus.cc:697] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [term 3 LEADER]: Becoming Leader. State: Replica: 098e2bd1d13041a08e2edee9dddf9b0b, State: Running, Role: LEADER
I20260812 19:53:30.447046 30636 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:30.447166 30642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:30.449487 30653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 098e2bd1d13041a08e2edee9dddf9b0b. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:30.449858 30653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:30.450368 30636 webserver.cc:533] Webserver started at http://127.25.14.1:39721/ using document root <none> and password file <none>
I20260812 19:53:30.450637 30657 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:30.451293 30636 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:30.451467 30636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:30.452739 30657 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:30.453552 30657 catalog_manager.cc:1295] Loaded cluster ID: 138cf4e2e0684b5580a0de4c6ee31bdb
I20260812 19:53:30.453683 30657 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:30.454862 30657 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:30.455672 30657 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b: Loaded TSK: 0
I20260812 19:53:30.456749 30657 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:30.458041 30636 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:30.461288 30659 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:30.462966 30636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:30.463164 30636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:30.463974 30636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:30.497339 30636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:30.498508 30636 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:30.498886 30636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:30.500171 30636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:30.501407 30665 raft_consensus.cc:493] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6de59a74a53c4c1089c1cc0d72e0c945)
I20260812 19:53:30.501674 30665 raft_consensus.cc:515] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } }
I20260812 19:53:30.502681 30636 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:30.502828 30636 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:30.503022 30636 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:30.503094 30636 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:30.504146 30665 leader_election.cc:290] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6de59a74a53c4c1089c1cc0d72e0c945 (127.25.14.62:35723), 098e2bd1d13041a08e2edee9dddf9b0b (127.25.14.61:41243)
I20260812 19:53:30.505091 29920 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" is_pre_election: true
I20260812 19:53:30.512666 30583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" is_pre_election: true
I20260812 19:53:30.513188 30583 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9b7e08ad9824c748fa847dbe0eb0f21 in current term 3: Already voted for candidate 098e2bd1d13041a08e2edee9dddf9b0b in this term.
I20260812 19:53:30.514277 29985 leader_election.cc:304] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b7e08ad9824c748fa847dbe0eb0f21; no voters: 098e2bd1d13041a08e2edee9dddf9b0b, 6de59a74a53c4c1089c1cc0d72e0c945
I20260812 19:53:30.514753 30665 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:53:30.517608 30665 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:53:30.544510 30636 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40247
I20260812 19:53:30.544697 30784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40247 every 8 connection(s)
I20260812 19:53:30.546361 30636 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:30.555198 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30636
I20260812 19:53:30.555546 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30636
I20260812 19:53:30.569056 30785 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:30.569566 30785 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:30.570984 30785 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:30.572041 30787 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:30.572216 30787 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:30.572576 30786 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:30.572764 30786 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:30.572971 30787 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:30.573282 30004 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:30.573350 30786 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:30.574534 30571 ts_manager.cc:194] Registered new tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:30.575819 29910 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:30.578362 29910 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:44905
I20260812 19:53:30.580201 30777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:30.644119 30374 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:30.644404 30374 heartbeater.cc:499] Master 127.25.14.62:35723 was elected leader, sending a full tablet report...
I20260812 19:53:30.645512 30571 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b89a527fca624c3e8a53b5309ada479c" instance_seqno: 1786564408589461) as {username='slave'} at 127.25.14.2:35685; Asking this server to re-register.
I20260812 19:53:30.646319 30374 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:30.647537 30571 ts_manager.cc:194] Registered new tserver with Master: b89a527fca624c3e8a53b5309ada479c (127.25.14.2:42125)
I20260812 19:53:30.674356 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30636
I20260812 19:53:30.702082 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40247
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=39721
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260812 19:53:31.009225 30014 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [term 3 FOLLOWER]: Refusing update from remote peer 098e2bd1d13041a08e2edee9dddf9b0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:53:31.010253 30798 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:53:31.012763 30665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 098e2bd1d13041a08e2edee9dddf9b0b. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.013113 30665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:31.016217 30665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.016472 30665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b7e08ad9824c748fa847dbe0eb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:31.016492 30800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.016722 30800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:31.016976 30583 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [term 3 FOLLOWER]: Refusing update from remote peer 098e2bd1d13041a08e2edee9dddf9b0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:53:31.018105 30800 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:53:31.022352 30807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.022677 30807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:31.023307 30804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 098e2bd1d13041a08e2edee9dddf9b0b. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.023626 30804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de59a74a53c4c1089c1cc0d72e0c945 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:31.096961 30797 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:31.097319 30797 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:31.097404 30797 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:31.097465 30797 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:31.107120 30797 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:31.107334 30797 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:31.118572 30797 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40247
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39721
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:31.120357 30797 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:31.122545 30797 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:31.134212 30812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:31.134256 30813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:31.134392 30816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:31.135847 30797 server_base.cc:1061] running on GCE node
I20260812 19:53:31.136591 30797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:31.137719 30797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:31.139024 30797 hybrid_clock.cc:648] HybridClock initialized: now 1786564411138927 us; error 66 us; skew 500 ppm
I20260812 19:53:31.139392 30797 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:31.141817 30797 webserver.cc:533] Webserver started at http://127.25.14.1:39721/ using document root <none> and password file <none>
I20260812 19:53:31.142661 30797 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:31.142830 30797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:31.148809 30797 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 19:53:31.152066 30822 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:31.153730 30797 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:31.153949 30797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:31.154822 30797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:31.170197 30797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:31.171185 30797 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:31.171492 30797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:31.172556 30797 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:31.174460 30797 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:31.174576 30797 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:31.174716 30797 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:31.174805 30797 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:31.175189 30797 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:31.175310 30797 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:31.175410 30797 fault_injection.cc:59] injecting fault for kudu (pid 30797): FLAGS_fault_before_start (process will exit)
I20260812 19:53:31.198791 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40247
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--webserver_port=39721
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:31.446816 30800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de59a74a53c4c1089c1cc0d72e0c945" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35723 } } peers { permanent_uuid: "098e2bd1d13041a08e2edee9dddf9b0b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41243 } } peers { permanent_uuid: "f9b7e08ad9824c748fa847dbe0eb0f21" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35609 } } }
I20260812 19:53:31.447115 30800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098e2bd1d13041a08e2edee9dddf9b0b [sys.catalog]: This master's current role is: LEADER
W20260812 19:53:31.574620 30830 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:31.574965 30830 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:31.575101 30830 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:31.583925 30830 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:31.584164 30830 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:31.595161 30830 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40247
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39721
--tserver_master_addrs=127.25.14.62:35723,127.25.14.61:41243,127.25.14.60:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:31.597146 30830 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:31.599315 30830 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:31.610271 30838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:31.610626 30840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:31.610249 30837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:31.611213 30830 server_base.cc:1061] running on GCE node
I20260812 19:53:31.611907 30830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:31.612957 30830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:31.614203 30830 hybrid_clock.cc:648] HybridClock initialized: now 1786564411614130 us; error 70 us; skew 500 ppm
I20260812 19:53:31.614550 30830 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:31.616986 30830 webserver.cc:533] Webserver started at http://127.25.14.1:39721/ using document root <none> and password file <none>
I20260812 19:53:31.617969 30830 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:31.618127 30830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:31.624003 30830 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 19:53:31.627032 30848 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:31.628540 30830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:31.628739 30830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "34c8fb2800ec4906b462f410145cb3f6"
format_stamp: "Formatted at 2026-08-12 19:53:20 on dist-test-slave-p3t5"
I20260812 19:53:31.629536 30830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:31.639560 30375 heartbeater.cc:499] Master 127.25.14.61:41243 was elected leader, sending a full tablet report...
I20260812 19:53:31.664326 30830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:31.665298 30830 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:31.665663 30830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:31.666709 30830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:31.668557 30830 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:31.668660 30830 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:31.668826 30830 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:31.668903 30830 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:31.706060 30830 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40247
I20260812 19:53:31.706120 30971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40247 every 8 connection(s)
I20260812 19:53:31.707810 30830 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:31.716163 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30830
I20260812 19:53:31.716615 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30830
I20260812 19:53:31.764151 30972 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35609
I20260812 19:53:31.764660 30972 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:31.765821 30972 heartbeater.cc:507] Master 127.25.14.60:35609 requested a full tablet report, sending...
I20260812 19:53:31.768127 30004 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:31.771019 30973 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35723
I20260812 19:53:31.771204 30973 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:31.771850 30973 heartbeater.cc:507] Master 127.25.14.62:35723 requested a full tablet report, sending...
I20260812 19:53:31.772477 30975 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41243
I20260812 19:53:31.772670 30975 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:31.773285 30975 heartbeater.cc:507] Master 127.25.14.61:41243 requested a full tablet report, sending...
I20260812 19:53:31.773510 30571 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:31.774411 29910 ts_manager.cc:194] Re-registered known tserver with Master: 34c8fb2800ec4906b462f410145cb3f6 (127.25.14.1:40247)
I20260812 19:53:31.775316 29910 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:43851
I20260812 19:53:31.775573 30967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:31.872581 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30830
I20260812 19:53:31.898020 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30235
I20260812 19:53:31.907639 30368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:32.001092 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30235
I20260812 19:53:32.025269 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30381
I20260812 19:53:32.033835 30520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:32.123273 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30381
I20260812 19:53:32.148703 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30538
I20260812 19:53:32.155391 30605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:32.268261 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30538
I20260812 19:53:32.298596 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29860
I20260812 19:53:32.300047 29942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:32.412911 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29860
I20260812 19:53:32.441018 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29949
I20260812 19:53:32.442270 30034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:32.551190 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29949
2026-08-12T19:53:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/1 (14428 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/2
2026-08-12T19:53:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:32Z Disabled control of system clock
2026-08-12T19:53:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:32Z Disabled control of system clock
2026-08-12T19:53:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:32Z Disabled control of system clock
2026-08-12T19:53:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:32Z Disabled control of system clock
2026-08-12T19:53:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:32Z Disabled control of system clock
I20260812 19:53:32.770819 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
W20260812 19:53:33.149044 31006 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:33.149420 31006 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:33.149556 31006 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:33.158327 31006 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:33.158429 31006 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:33.158526 31006 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:33.158627 31006 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:33.169360 31006 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:33.171241 31006 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:33.173220 31006 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:33.182296 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:33.182359 31012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:33.183132 31014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:33.184427 31006 server_base.cc:1061] running on GCE node
I20260812 19:53:33.185350 31006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:33.187297 31006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:33.190721 31006 hybrid_clock.cc:648] HybridClock initialized: now 1786564413190474 us; error 217 us; skew 500 ppm
I20260812 19:53:33.191243 31006 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:33.193842 31006 webserver.cc:533] Webserver started at http://127.25.14.62:33927/ using document root <none> and password file <none>
I20260812 19:53:33.194697 31006 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:33.194855 31006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:33.195251 31006 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:33.197934 31006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.198828 31006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.205147 31006 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:53:33.209189 31023 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:33.210930 31006 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:33.211167 31006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.211500 31006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:33.226445 31006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:33.227447 31006 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:33.227761 31006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:33.249477 31006 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:45955
I20260812 19:53:33.249503 31079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:45955 every 8 connection(s)
I20260812 19:53:33.251220 31006 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:33.256270 31080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:33.258378 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31006
I20260812 19:53:33.258636 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:33.260136 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40565
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
I20260812 19:53:33.265723 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:33.285858 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:33.287395 31028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
W20260812 19:53:33.289431 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
I20260812 19:53:33.338884 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } attempt: 1
W20260812 19:53:33.341290 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
I20260812 19:53:33.409643 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } attempt: 2
W20260812 19:53:33.412302 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
I20260812 19:53:33.503698 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } attempt: 3
W20260812 19:53:33.506359 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
W20260812 19:53:33.634805 31082 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:33.635203 31082 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:33.635288 31082 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:33.644919 31082 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:33.645023 31082 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:33.645081 31082 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:33.645123 31082 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:33.649694 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } attempt: 4
W20260812 19:53:33.652012 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111)
I20260812 19:53:33.656219 31082 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:33.658013 31082 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:33.659931 31082 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:33.668785 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:33.668948 31090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:33.669699 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:33.670111 31082 server_base.cc:1061] running on GCE node
I20260812 19:53:33.671002 31082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:33.672709 31082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:33.674055 31082 hybrid_clock.cc:648] HybridClock initialized: now 1786564413673906 us; error 113 us; skew 500 ppm
I20260812 19:53:33.674438 31082 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:33.676877 31082 webserver.cc:533] Webserver started at http://127.25.14.61:44071/ using document root <none> and password file <none>
I20260812 19:53:33.677760 31082 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:33.677891 31082 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:33.678298 31082 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:33.680886 31082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "153a45b172f642818c4340f35e4d8ef7"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.681784 31082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "153a45b172f642818c4340f35e4d8ef7"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.687502 31082 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:53:33.691299 31100 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:33.692807 31082 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:33.693004 31082 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "153a45b172f642818c4340f35e4d8ef7"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:33.693261 31082 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:33.718395 31082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:33.719367 31082 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:33.719689 31082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:33.741921 31082 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:40565
I20260812 19:53:33.741942 31154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:40565 every 8 connection(s)
I20260812 19:53:33.743655 31082 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:33.748381 31155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:33.750808 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31082
I20260812 19:53:33.751012 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:33.752468 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44381
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
I20260812 19:53:33.757354 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:33.773319 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:33.785145 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:33.786907 31104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
W20260812 19:53:33.788758 31155 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:33.838199 31155 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 1
W20260812 19:53:33.840457 31155 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:33.908751 31155 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 2
W20260812 19:53:33.911077 31155 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:33.951336 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } attempt: 5
I20260812 19:53:33.958576 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:53:33.960511 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:34.002389 31155 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 3
W20260812 19:53:34.004670 31155 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:34.011749 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 1
W20260812 19:53:34.013957 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:34.082242 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 2
W20260812 19:53:34.084666 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
W20260812 19:53:34.129433 31157 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:34.129953 31157 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:34.130100 31157 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:34.138670 31157 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:34.138777 31157 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:34.138904 31157 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:34.138989 31157 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:34.147948 31155 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 4
I20260812 19:53:34.149752 31157 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
W20260812 19:53:34.150203 31155 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:34.151592 31157 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:34.153481 31157 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:34.162467 31166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:34.162647 31169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:34.162846 31167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:34.163992 31157 server_base.cc:1061] running on GCE node
I20260812 19:53:34.164834 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:34.166762 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:34.167985 31157 hybrid_clock.cc:648] HybridClock initialized: now 1786564414167853 us; error 94 us; skew 500 ppm
I20260812 19:53:34.168419 31157 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:34.170996 31157 webserver.cc:533] Webserver started at http://127.25.14.60:43637/ using document root <none> and password file <none>
I20260812 19:53:34.171845 31157 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:34.171978 31157 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:34.172386 31157 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:34.174949 31157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "b3fa8e711a5c47b2a10028be5d057ad3"
format_stamp: "Formatted at 2026-08-12 19:53:34 on dist-test-slave-p3t5"
I20260812 19:53:34.175772 31157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "b3fa8e711a5c47b2a10028be5d057ad3"
format_stamp: "Formatted at 2026-08-12 19:53:34 on dist-test-slave-p3t5"
I20260812 19:53:34.181804 31157 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 19:53:34.185705 31177 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:34.187284 31157 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:34.187512 31157 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "b3fa8e711a5c47b2a10028be5d057ad3"
format_stamp: "Formatted at 2026-08-12 19:53:34 on dist-test-slave-p3t5"
I20260812 19:53:34.187794 31157 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:34.190996 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 3
W20260812 19:53:34.193109 31080 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111)
I20260812 19:53:34.210181 31157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:34.211198 31157 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:34.211465 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:34.233196 31157 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:44381
I20260812 19:53:34.233218 31231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:44381 every 8 connection(s)
I20260812 19:53:34.234900 31157 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:34.239607 31232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:53:34.241453 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31157
I20260812 19:53:34.241714 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:34.248539 31232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:34.265020 31232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:34.272186 31232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:53:34.283030 31232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Bootstrap starting.
I20260812 19:53:34.287241 31232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:34.288462 31232 log.cc:826] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:34.291249 31232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: No bootstrap required, opened a new log
I20260812 19:53:34.297057 31232 raft_consensus.cc:359] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.297376 31232 raft_consensus.cc:385] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:34.297533 31232 raft_consensus.cc:740] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fa8e711a5c47b2a10028be5d057ad3, State: Initialized, Role: FOLLOWER
I20260812 19:53:34.298254 31232 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.299317 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.299630 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.300422 31232 sys_catalog.cc:565] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:53:34.311686 31252 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:34.311861 31252 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:53:34.308655 31157 master_runner.cc:187] Error getting master registration for 127.25.14.62:45955: OK, instance_id { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" instance_seqno: 1786564413243539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:34.313464 31157 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:34.334892 31239 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:34.335222 31239 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.337373 31239 leader_election.cc:290] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955), 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565)
I20260812 19:53:34.370378 31080 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 4
I20260812 19:53:34.384114 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap starting.
I20260812 19:53:34.388942 31080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:34.390560 31080 log.cc:826] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:34.394102 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: No bootstrap required, opened a new log
I20260812 19:53:34.402747 31080 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.403223 31080 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:34.403381 31080 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef649735d074b13845c3ee20ee3af82, State: Initialized, Role: FOLLOWER
I20260812 19:53:34.404222 31080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.406132 31255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.407003 31255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.407335 31080 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:34.407413 31053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef649735d074b13845c3ee20ee3af82" is_pre_election: true
I20260812 19:53:34.407949 31053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3fa8e711a5c47b2a10028be5d057ad3 in term 0.
I20260812 19:53:34.409029 31180 leader_election.cc:304] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3; no voters:
I20260812 19:53:34.409663 31239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:53:34.409840 31239 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:34.410012 31239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:34.415014 31239 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.419610 31053 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef649735d074b13845c3ee20ee3af82"
I20260812 19:53:34.419842 31053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:34.423758 31053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3fa8e711a5c47b2a10028be5d057ad3 in term 1.
I20260812 19:53:34.424381 31180 leader_election.cc:304] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3; no voters:
I20260812 19:53:34.426365 31239 leader_election.cc:290] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955), 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565)
I20260812 19:53:34.427116 31239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:53:34.427681 31239 raft_consensus.cc:697] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 LEADER]: Becoming Leader. State: Replica: b3fa8e711a5c47b2a10028be5d057ad3, State: Running, Role: LEADER
I20260812 19:53:34.428503 31239 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.420400 31006 master_runner.cc:187] Error getting master registration for 127.25.14.61:40565: OK, instance_id { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" instance_seqno: 1786564413735883 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:53:34.430140 31006 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:34.434815 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.439486 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:34.441800 31275 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:34.444353 31275 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:34.449436 31155 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } attempt: 5
I20260812 19:53:34.455765 31053 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Refusing update from remote peer b3fa8e711a5c47b2a10028be5d057ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:53:34.456774 31270 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:53:34.456916 31270 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:53:34.463647 31183 tablet.cc:2406] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:53:34.465870 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Bootstrap starting.
I20260812 19:53:34.465838 31273 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:34.474156 31155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Neither blocks nor log segments found. Creating new log.
I20260812 19:53:34.478001 31155 log.cc:826] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Log is configured to *not* fsync() on all Append() calls
W20260812 19:53:34.480954 31030 tablet.cc:2406] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:53:34.491698 31255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.492097 31255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.493719 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: No bootstrap required, opened a new log
I20260812 19:53:34.500653 31276 mvcc.cc:204] Tried to move back new op lower bound from 7317767841593442304 to 7317767841510338560. Current Snapshot: MvccSnapshot[applied={T|T < 7317767841593442304}]
I20260812 19:53:34.509258 31155 raft_consensus.cc:359] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.509701 31155 raft_consensus.cc:385] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:53:34.509843 31155 raft_consensus.cc:740] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153a45b172f642818c4340f35e4d8ef7, State: Initialized, Role: FOLLOWER
I20260812 19:53:34.510803 31155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:34.517928 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.518231 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:34.521487 31275 catalog_manager.cc:1383] Generated new cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:34.521607 31275 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:34.534708 31255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.534987 31255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.543402 31282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.543795 31282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.545202 31155 sys_catalog.cc:565] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:34.545727 31127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:53:34.549769 31129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7" is_pre_election: true
I20260812 19:53:34.545208 31128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7"
I20260812 19:53:34.556164 31127 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 1 FOLLOWER]: Refusing update from remote peer b3fa8e711a5c47b2a10028be5d057ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:53:34.557458 31273 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:53:34.594678 31275 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:53:34.599722 31275 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:34.611217 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.611655 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.611802 31282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.612016 31282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:34.614215 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:34.614523 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:34.628505 31275 catalog_manager.cc:6092] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Generated new TSK 0
I20260812 19:53:34.629801 31275 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:34.645511 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:34.662547 31300 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:34.662678 31300 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: loading cluster ID for follower catalog manager: success
I20260812 19:53:34.666963 31300 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: acquiring CA information for follower catalog manager: success
I20260812 19:53:34.669834 31300 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:53:35.228372 31302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:35.228749 31302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:35.228826 31302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:35.237902 31302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:35.238121 31302 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:35.248586 31302 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:35.250833 31302 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:35.253284 31302 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:35.266119 31307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:35.266141 31308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:35.267051 31310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:35.268342 31302 server_base.cc:1061] running on GCE node
I20260812 19:53:35.269070 31302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:35.270504 31302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:35.271910 31302 hybrid_clock.cc:648] HybridClock initialized: now 1786564415271714 us; error 157 us; skew 500 ppm
I20260812 19:53:35.272296 31302 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:35.275076 31302 webserver.cc:533] Webserver started at http://127.25.14.1:39295/ using document root <none> and password file <none>
I20260812 19:53:35.276021 31302 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:35.276120 31302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:35.276432 31302 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:35.279001 31302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.279770 31302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.285516 31302 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:53:35.289129 31316 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.290704 31302 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 19:53:35.290863 31302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.291075 31302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:35.319759 31302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:35.320809 31302 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:35.321147 31302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:35.322425 31302 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:35.324426 31302 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:35.324543 31302 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.324703 31302 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:35.324805 31302 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.361246 31302 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43777
I20260812 19:53:35.361351 31446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43777 every 8 connection(s)
I20260812 19:53:35.362974 31302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:35.363315 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31302
I20260812 19:53:35.363551 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:35.372234 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:35.406033 31449 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:35.406755 31449 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.407909 31449 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:35.406201 31448 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:35.409479 31448 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.410168 31448 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:35.410529 31447 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:35.410660 31447 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.411058 31116 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:35.411222 31447 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:35.412269 31041 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:35.412998 31195 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:35.414794 31195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:44851
I20260812 19:53:35.458952 31270 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:35.459100 31270 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: loading cluster ID for follower catalog manager: success
I20260812 19:53:35.461898 31270 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: acquiring CA information for follower catalog manager: success
I20260812 19:53:35.463976 31270 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:53:35.753751 31458 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:35.754128 31458 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:35.754292 31458 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:35.763499 31458 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:35.763732 31458 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:35.774505 31458 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:35.776368 31458 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:35.778581 31458 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:35.790283 31464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:35.790414 31463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:35.791181 31466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:35.791522 31458 server_base.cc:1061] running on GCE node
I20260812 19:53:35.792233 31458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:35.793974 31458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:35.795348 31458 hybrid_clock.cc:648] HybridClock initialized: now 1786564415795185 us; error 121 us; skew 500 ppm
I20260812 19:53:35.795711 31458 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:35.798599 31458 webserver.cc:533] Webserver started at http://127.25.14.2:41087/ using document root <none> and password file <none>
I20260812 19:53:35.799500 31458 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:35.799640 31458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:35.800037 31458 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:35.802910 31458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "ccc31e8c1308402d96f39e5e8d8eb43e"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.803823 31458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "ccc31e8c1308402d96f39e5e8d8eb43e"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.810658 31458 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 19:53:35.814668 31472 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.816270 31458 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:35.816514 31458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "ccc31e8c1308402d96f39e5e8d8eb43e"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:35.816886 31458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:35.837802 31458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:35.838829 31458 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:35.839147 31458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:35.840222 31458 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:35.842163 31458 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:35.842279 31458 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.842435 31458 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:35.842525 31458 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:35.880045 31458 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:33635
I20260812 19:53:35.880162 31595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:33635 every 8 connection(s)
I20260812 19:53:35.881595 31458 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:35.890601 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31458
I20260812 19:53:35.890849 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:35.895627 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:35.902685 31598 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:35.903013 31597 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:35.903194 31597 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.903192 31598 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.904763 31597 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:35.904819 31598 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:35.905757 31596 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:35.905920 31596 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:35.906479 31596 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:35.906940 31116 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:33635)
I20260812 19:53:35.906952 31041 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:33635)
I20260812 19:53:35.907526 31195 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:33635)
I20260812 19:53:35.908556 31195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:42897
W20260812 19:53:36.273830 31604 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:36.274345 31604 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:36.274495 31604 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:36.283602 31604 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:36.283854 31604 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:36.295393 31604 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:36.297322 31604 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:36.299507 31604 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:36.311309 31610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:36.311287 31609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:36.312031 31612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:36.312276 31604 server_base.cc:1061] running on GCE node
I20260812 19:53:36.312995 31604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:36.314348 31604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:36.315670 31604 hybrid_clock.cc:648] HybridClock initialized: now 1786564416315505 us; error 119 us; skew 500 ppm
I20260812 19:53:36.316027 31604 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:36.318686 31604 webserver.cc:533] Webserver started at http://127.25.14.3:41465/ using document root <none> and password file <none>
I20260812 19:53:36.319572 31604 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:36.319676 31604 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:36.320017 31604 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:53:36.322760 31604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "68476b355405493eaad1c0bba70a70f2"
format_stamp: "Formatted at 2026-08-12 19:53:36 on dist-test-slave-p3t5"
I20260812 19:53:36.323613 31604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "68476b355405493eaad1c0bba70a70f2"
format_stamp: "Formatted at 2026-08-12 19:53:36 on dist-test-slave-p3t5"
I20260812 19:53:36.329720 31604 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 19:53:36.333446 31618 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:36.335078 31604 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:36.335249 31604 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "68476b355405493eaad1c0bba70a70f2"
format_stamp: "Formatted at 2026-08-12 19:53:36 on dist-test-slave-p3t5"
I20260812 19:53:36.335462 31604 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:36.354079 31604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:36.355091 31604 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:36.355350 31604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:36.356387 31604 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:36.358268 31604 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:36.358381 31604 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:36.358466 31604 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:36.358520 31604 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:36.396026 31604 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:33183
I20260812 19:53:36.396080 31744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:33183 every 8 connection(s)
I20260812 19:53:36.397846 31604 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:36.407907 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31604
I20260812 19:53:36.408177 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:36.417603 31447 heartbeater.cc:499] Master 127.25.14.60:44381 was elected leader, sending a full tablet report...
I20260812 19:53:36.418731 31745 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:36.419049 31747 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:36.419185 31745 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:36.419225 31747 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:36.420125 31746 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:36.420327 31746 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:36.420624 31747 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:36.420650 31745 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:36.421003 31746 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:36.422819 31196 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:33183)
I20260812 19:53:36.422976 31119 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:33183)
I20260812 19:53:36.423992 31041 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:33183)
I20260812 19:53:36.424067 31196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:34509
I20260812 19:53:36.441304 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:36.449256 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:36.449443 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31006
I20260812 19:53:36.450402 31073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:36.562848 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31006
I20260812 19:53:36.592666 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--webserver_interface=127.25.14.62
--webserver_port=33927
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
W20260812 19:53:36.910391 31597 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:45955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:45955: connect: Connection refused (error 111)
I20260812 19:53:36.910807 31596 heartbeater.cc:499] Master 127.25.14.60:44381 was elected leader, sending a full tablet report...
W20260812 19:53:36.918612 31180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:45955: connect: Connection refused (error 111)
W20260812 19:53:36.920291 31180 consensus_peers.cc:597] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 -> Peer 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955): Couldn't send request to peer 3ef649735d074b13845c3ee20ee3af82. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:45955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:53:36.973277 31755 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:36.973701 31755 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:36.973840 31755 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:36.982422 31755 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:36.982553 31755 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:36.982662 31755 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:36.982743 31755 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:36.993357 31755 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=33927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:36.995169 31755 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:36.997254 31755 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:37.006939 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:37.007804 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:37.007992 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:37.008576 31755 server_base.cc:1061] running on GCE node
I20260812 19:53:37.009429 31755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:37.011359 31755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:37.012748 31755 hybrid_clock.cc:648] HybridClock initialized: now 1786564417012559 us; error 149 us; skew 500 ppm
I20260812 19:53:37.013168 31755 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:37.015684 31755 webserver.cc:533] Webserver started at http://127.25.14.62:33927/ using document root <none> and password file <none>
I20260812 19:53:37.016533 31755 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:37.016688 31755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:37.022755 31755 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:37.026058 31772 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:37.027593 31755 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:53:37.027786 31755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:37.028584 31755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:37.057428 31755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:37.058480 31755 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:37.058815 31755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:37.080883 31755 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:45955
I20260812 19:53:37.080907 31826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:45955 every 8 connection(s)
I20260812 19:53:37.082836 31755 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:37.089447 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31755
I20260812 19:53:37.089897 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31302
I20260812 19:53:37.091187 31827 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:37.095156 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap starting.
I20260812 19:53:37.105288 31442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:37.115023 31827 log.cc:826] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:37.124667 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:37.125291 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap complete.
I20260812 19:53:37.133088 31827 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:37.133982 31827 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef649735d074b13845c3ee20ee3af82, State: Initialized, Role: FOLLOWER
I20260812 19:53:37.134872 31827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:37.135829 31833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:37.136142 31833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:37.137072 31827 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:37.149211 31846 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:37.150647 31846 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: loading cluster ID for follower catalog manager: success
I20260812 19:53:37.156003 31846 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: acquiring CA information for follower catalog manager: success
I20260812 19:53:37.159143 31846 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:37.225512 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31302
I20260812 19:53:37.258668 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43777
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=39295
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:37.356647 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:37.356969 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:37.369586 31833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:37.369971 31833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:37.426494 31745 heartbeater.cc:499] Master 127.25.14.60:44381 was elected leader, sending a full tablet report...
I20260812 19:53:37.439296 31790 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "68476b355405493eaad1c0bba70a70f2" instance_seqno: 1786564416387319) as {username='slave'} at 127.25.14.3:55939; Asking this server to re-register.
I20260812 19:53:37.440356 31746 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:37.442431 31790 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:33183)
W20260812 19:53:37.668282 31852 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:37.668655 31852 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:37.668740 31852 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:37.677598 31852 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:37.677873 31852 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:37.689076 31852 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43777
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39295
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:37.690958 31852 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:37.693075 31852 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:37.704705 31867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:37.704687 31866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:37.705453 31870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:37.706279 31852 server_base.cc:1061] running on GCE node
I20260812 19:53:37.706908 31852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:37.708257 31852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:37.709494 31852 hybrid_clock.cc:648] HybridClock initialized: now 1786564417709342 us; error 116 us; skew 500 ppm
I20260812 19:53:37.709909 31852 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:37.712101 31852 webserver.cc:533] Webserver started at http://127.25.14.1:39295/ using document root <none> and password file <none>
I20260812 19:53:37.712873 31852 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:37.712996 31852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:37.718945 31852 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:53:37.722043 31876 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:37.723738 31852 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:53:37.723909 31852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:37.724654 31852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:37.759832 31852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:37.760800 31852 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:37.761091 31852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:37.762130 31852 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:37.763993 31852 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:37.764073 31852 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:37.764168 31852 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:37.764217 31852 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:37.801694 31852 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43777
I20260812 19:53:37.801750 32003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43777 every 8 connection(s)
I20260812 19:53:37.803418 31852 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:37.811995 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31852
I20260812 19:53:37.812366 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31852
I20260812 19:53:37.824429 32007 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:37.824443 32006 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:37.825412 32007 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:37.825449 32006 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:37.826596 31853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:37.826906 31853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:37.827033 32006 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:37.827068 32007 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:37.829099 31790 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:37.829486 31119 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:37.834038 32004 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:37.834187 32004 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:37.834726 32004 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:37.835956 31196 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:37.836975 31196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:37461
I20260812 19:53:37.843089 31998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:37.918946 31597 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:37.920042 31790 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ccc31e8c1308402d96f39e5e8d8eb43e" instance_seqno: 1786564415871496) as {username='slave'} at 127.25.14.2:39061; Asking this server to re-register.
I20260812 19:53:37.920626 31597 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:37.921939 31790 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:33635)
I20260812 19:53:37.935755 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31852
I20260812 19:53:37.961939 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43777
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=39295
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260812 19:53:38.335587 32018 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:38.335969 32018 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:38.336047 32018 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:38.336105 32018 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:38.344978 32018 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:38.345182 32018 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:38.356271 32018 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43777
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39295
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:38.358080 32018 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:38.360044 32018 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:38.370399 32023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:38.370460 32024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:38.370832 32027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:38.371359 32018 server_base.cc:1061] running on GCE node
I20260812 19:53:38.372045 32018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:38.373229 32018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:38.374567 32018 hybrid_clock.cc:648] HybridClock initialized: now 1786564418374394 us; error 119 us; skew 500 ppm
I20260812 19:53:38.374898 32018 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:38.377336 32018 webserver.cc:533] Webserver started at http://127.25.14.1:39295/ using document root <none> and password file <none>
I20260812 19:53:38.378229 32018 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:38.378396 32018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:38.384302 32018 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:53:38.387516 32035 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:38.389029 32018 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 19:53:38.389236 32018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:38.390089 32018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:38.411597 32018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:38.412629 32018 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:38.412962 32018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:38.414095 32018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:38.416014 32018 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:38.416148 32018 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:38.416277 32018 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:38.416373 32018 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:38.416782 32018 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:38.416895 32018 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:38.416991 32018 fault_injection.cc:59] injecting fault for kudu (pid 32018): FLAGS_fault_before_start (process will exit)
I20260812 19:53:38.448625 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43777
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=39295
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:53:38.830418 32043 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:38.830793 32043 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:38.830883 32043 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:38.839826 32043 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:38.840013 32043 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:38.850833 32043 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39601,127.25.14.21:43439,127.25.14.22:44343,127.25.14.23:36301,127.25.14.24:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43777
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39295
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:38.852583 32043 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:38.854650 32043 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:38.865337 32048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:38.865923 32051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:38.865356 32049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:38.866921 32043 server_base.cc:1061] running on GCE node
I20260812 19:53:38.867573 32043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:38.868804 32043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:38.872179 32043 hybrid_clock.cc:648] HybridClock initialized: now 1786564418872026 us; error 109 us; skew 500 ppm
I20260812 19:53:38.872562 32043 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:38.878347 32043 webserver.cc:533] Webserver started at http://127.25.14.1:39295/ using document root <none> and password file <none>
I20260812 19:53:38.879190 32043 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:38.879324 32043 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:38.886623 32043 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:53:38.889817 32062 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:38.891353 32043 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:53:38.891549 32043 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:38.892289 32043 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:38.918185 32043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:38.919135 32043 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:38.919451 32043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:38.920459 32043 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:38.922338 32043 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:38.922413 32043 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:38.922570 32043 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:38.922621 32043 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:38.957659 32043 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43777
I20260812 19:53:38.957712 32191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43777 every 8 connection(s)
I20260812 19:53:38.959334 32043 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:38.965996 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32043
I20260812 19:53:38.966418 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32043
I20260812 19:53:38.998303 32192 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:38.998760 32192 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:39.000314 32192 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:39.002921 32195 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:39.003135 32194 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:39.003154 32195 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:39.003337 32194 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:39.004135 32194 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:39.004160 32195 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:39.005899 31790 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:39.007000 31119 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:39.007541 32187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.008373 31196 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:43777)
I20260812 19:53:39.009919 31196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41995
I20260812 19:53:39.123018 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32043
I20260812 19:53:39.148306 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31458
I20260812 19:53:39.157196 31589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.247573 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31458
I20260812 19:53:39.272329 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31604
I20260812 19:53:39.281193 31740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.375012 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31604
I20260812 19:53:39.399528 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31755
I20260812 19:53:39.406553 31821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.515266 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31755
I20260812 19:53:39.543638 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31082
I20260812 19:53:39.545043 31150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.655893 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31082
I20260812 19:53:39.684301 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31157
I20260812 19:53:39.685549 31227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:39.798802 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31157
2026-08-12T19:53:39Z chronyd exiting
2026-08-12T19:53:39Z chronyd exiting
2026-08-12T19:53:39Z chronyd exiting
2026-08-12T19:53:39Z chronyd exiting
2026-08-12T19:53:39Z chronyd exiting
2026-08-12T19:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:39Z Disabled control of system clock
2026-08-12T19:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:39Z Disabled control of system clock
2026-08-12T19:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:39Z Disabled control of system clock
2026-08-12T19:53:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:40Z Disabled control of system clock
2026-08-12T19:53:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:53:40Z Disabled control of system clock
I20260812 19:53:40.052635 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
W20260812 19:53:40.415699 32228 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:40.416162 32228 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:40.416308 32228 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:40.425009 32228 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:40.425177 32228 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:40.425285 32228 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:40.425372 32228 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:40.436033 32228 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:40.437958 32228 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:40.440034 32228 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:40.449222 32234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:40.449393 32237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:40.449255 32235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:40.450645 32228 server_base.cc:1061] running on GCE node
I20260812 19:53:40.451550 32228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:40.453274 32228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:40.454538 32228 hybrid_clock.cc:648] HybridClock initialized: now 1786564420454383 us; error 120 us; skew 500 ppm
I20260812 19:53:40.454944 32228 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:40.457387 32228 webserver.cc:533] Webserver started at http://127.25.14.62:34097/ using document root <none> and password file <none>
I20260812 19:53:40.458282 32228 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:40.458431 32228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:40.464166 32228 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:53:40.467767 32243 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:40.474466 32228 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.001s sys 0.001s
I20260812 19:53:40.474737 32228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:40.475530 32228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:40.525094 32228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:40.526196 32228 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:40.526513 32228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:40.548880 32228 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:45955
I20260812 19:53:40.548923 32301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:45955 every 8 connection(s)
I20260812 19:53:40.550802 32228 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:40.558892 32302 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:40.558940 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32228
I20260812 19:53:40.559137 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:53:40.560599 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40565
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
I20260812 19:53:40.564471 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap starting.
I20260812 19:53:40.582257 32302 log.cc:826] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:40.591478 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:40.592057 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap complete.
I20260812 19:53:40.599368 32302 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:40.600123 32302 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef649735d074b13845c3ee20ee3af82, State: Initialized, Role: FOLLOWER
I20260812 19:53:40.600919 32302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:40.601871 32307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:40.602178 32307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:40.602978 32302 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:40.614022 32321 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:40.614162 32321 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: loading cluster ID for follower catalog manager: success
I20260812 19:53:40.619827 32321 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: acquiring CA information for follower catalog manager: success
I20260812 19:53:40.623579 32321 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:40.640162 32228 master_runner.cc:187] Error getting master registration for 127.25.14.61:40565: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40565: connect: Connection refused (error 111),
I20260812 19:53:40.646194 32228 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:40.958978 32305 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:40.959328 32305 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:40.959440 32305 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:40.968447 32305 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:40.968557 32305 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:40.968644 32305 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:40.968734 32305 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:40.980435 32305 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:40.982445 32305 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:40.984611 32305 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:40.993830 32331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:40.993844 32332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:40.994918 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:40.996083 32305 server_base.cc:1061] running on GCE node
I20260812 19:53:40.997059 32305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:40.998924 32305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:41.002350 32305 hybrid_clock.cc:648] HybridClock initialized: now 1786564421002168 us; error 130 us; skew 500 ppm
I20260812 19:53:41.002810 32305 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:41.005432 32305 webserver.cc:533] Webserver started at http://127.25.14.61:32917/ using document root <none> and password file <none>
I20260812 19:53:41.006425 32305 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:41.006603 32305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:41.013051 32305 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:53:41.016728 32341 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:41.018471 32305 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:41.018707 32305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "153a45b172f642818c4340f35e4d8ef7"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:41.019520 32305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:41.039579 32305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:41.040557 32305 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:41.040885 32305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:41.063283 32305 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:40565
I20260812 19:53:41.063333 32396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:40565 every 8 connection(s)
I20260812 19:53:41.065136 32305 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:53:41.068303 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32305
I20260812 19:53:41.068549 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:53:41.069522 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44381
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
I20260812 19:53:41.072929 32397 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:41.076416 32397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Bootstrap starting.
I20260812 19:53:41.097987 32397 log.cc:826] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:41.107997 32397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:41.108697 32397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: Bootstrap complete.
I20260812 19:53:41.116186 32397 raft_consensus.cc:359] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:41.116927 32397 raft_consensus.cc:740] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153a45b172f642818c4340f35e4d8ef7, State: Initialized, Role: FOLLOWER
I20260812 19:53:41.117739 32397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:41.118716 32403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:41.119045 32403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:41.119887 32397 sys_catalog.cc:565] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:41.131304 32414 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:41.131541 32414 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: loading cluster ID for follower catalog manager: success
I20260812 19:53:41.135931 32414 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: acquiring CA information for follower catalog manager: success
I20260812 19:53:41.138679 32414 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:41.157179 32305 master_runner.cc:187] Error getting master registration for 127.25.14.60:44381: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44381: connect: Connection refused (error 111),
I20260812 19:53:41.161115 32305 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:53:41.486212 32400 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:41.486608 32400 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:41.486759 32400 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:41.495631 32400 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:41.495764 32400 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:41.495872 32400 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:41.495959 32400 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:41.507360 32400 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:41.509258 32400 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:41.511338 32400 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:41.520638 32423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:41.520615 32424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:41.522683 32427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:41.523384 32400 server_base.cc:1061] running on GCE node
I20260812 19:53:41.524343 32400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:41.526289 32400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:41.527622 32400 hybrid_clock.cc:648] HybridClock initialized: now 1786564421527437 us; error 152 us; skew 500 ppm
I20260812 19:53:41.527999 32400 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:41.530591 32400 webserver.cc:533] Webserver started at http://127.25.14.60:45005/ using document root <none> and password file <none>
I20260812 19:53:41.531560 32400 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:41.531710 32400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:41.537796 32400 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:53:41.541066 32433 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:41.542676 32400 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:41.542867 32400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "b3fa8e711a5c47b2a10028be5d057ad3"
format_stamp: "Formatted at 2026-08-12 19:53:34 on dist-test-slave-p3t5"
I20260812 19:53:41.543594 32400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:41.594254 32400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:41.595238 32400 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:41.595556 32400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:41.617262 32400 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:44381
I20260812 19:53:41.617300 32492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:44381 every 8 connection(s)
I20260812 19:53:41.619154 32400 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:53:41.620971 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32400
I20260812 19:53:41.621230 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:53:41.629035 32493 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:41.634021 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Bootstrap starting.
I20260812 19:53:41.654040 32493 log.cc:826] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:41.663908 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:41.664659 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Bootstrap complete.
I20260812 19:53:41.671147 32493 raft_consensus.cc:359] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:41.671834 32493 raft_consensus.cc:740] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fa8e711a5c47b2a10028be5d057ad3, State: Initialized, Role: FOLLOWER
I20260812 19:53:41.672612 32493 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:41.673888 32499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:41.674192 32499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:41.675072 32493 sys_catalog.cc:565] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:41.686417 32510 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:41.686558 32510 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: loading cluster ID for follower catalog manager: success
I20260812 19:53:41.690099 32510 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: acquiring CA information for follower catalog manager: success
I20260812 19:53:41.692641 32510 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:41.711978 32400 master_runner.cc:264] No leader master found from master b3fa8e711a5c47b2a10028be5d057ad3
I20260812 19:53:41.712163 32400 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:42.415597 32513 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:53:42.415956 32513 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:42.418073 32513 leader_election.cc:290] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565), b3fa8e711a5c47b2a10028be5d057ad3 (127.25.14.60:44381)
I20260812 19:53:42.425997 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ef649735d074b13845c3ee20ee3af82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7" is_pre_election: true
I20260812 19:53:42.426738 32372 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef649735d074b13845c3ee20ee3af82 in term 1.
I20260812 19:53:42.427119 32465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ef649735d074b13845c3ee20ee3af82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" is_pre_election: true
I20260812 19:53:42.427523 32248 leader_election.cc:304] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153a45b172f642818c4340f35e4d8ef7, 3ef649735d074b13845c3ee20ee3af82; no voters:
I20260812 19:53:42.427714 32465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef649735d074b13845c3ee20ee3af82 in term 1.
I20260812 19:53:42.428018 32513 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:53:42.428215 32513 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:53:42.428352 32513 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:42.432320 32513 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:42.432991 32513 leader_election.cc:290] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [CANDIDATE]: Term 2 election: Requested vote from peers 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565), b3fa8e711a5c47b2a10028be5d057ad3 (127.25.14.60:44381)
I20260812 19:53:42.433619 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ef649735d074b13845c3ee20ee3af82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7"
I20260812 19:53:42.433772 32465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ef649735d074b13845c3ee20ee3af82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b3fa8e711a5c47b2a10028be5d057ad3"
I20260812 19:53:42.433879 32372 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:42.433996 32465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:53:42.439625 32465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef649735d074b13845c3ee20ee3af82 in term 2.
I20260812 19:53:42.439625 32372 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef649735d074b13845c3ee20ee3af82 in term 2.
I20260812 19:53:42.440250 32248 leader_election.cc:304] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3; no voters:
I20260812 19:53:42.440603 32513 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:53:42.441036 32513 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 LEADER]: Becoming Leader. State: Replica: 3ef649735d074b13845c3ee20ee3af82, State: Running, Role: LEADER
I20260812 19:53:42.441567 32513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:42.444588 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef649735d074b13845c3ee20ee3af82. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.444923 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:42.445685 32521 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:42.447763 32521 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:42.448936 32521 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:42.449043 32521 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:42.450453 32521 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:42.451191 32521 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Loaded TSK: 0
I20260812 19:53:42.452327 32521 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:42.492661 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:53:42.867156 32522 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:42.867570 32522 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:42.867709 32522 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:42.876394 32522 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:42.876660 32522 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:42.887804 32522 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:42.888132 32465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Refusing update from remote peer 3ef649735d074b13845c3ee20ee3af82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:53:42.889253 32518 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:42.889914 32522 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:42.892004 32522 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:53:42.894093 32526 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef649735d074b13845c3ee20ee3af82. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.894469 32526 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:42.895463 32513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.895742 32513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:42.896713 32372 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 2 FOLLOWER]: Refusing update from remote peer 3ef649735d074b13845c3ee20ee3af82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:53:42.898828 32526 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.899154 32526 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:42.899441 32529 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:53:42.905836 32536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.906322 32536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:42.906739 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef649735d074b13845c3ee20ee3af82. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:42.907136 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:42.908169 32534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:42.908205 32537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:42.910286 32539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:42.910959 32522 server_base.cc:1061] running on GCE node
I20260812 19:53:42.911629 32522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:42.912880 32522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:42.916244 32522 hybrid_clock.cc:648] HybridClock initialized: now 1786564422916066 us; error 129 us; skew 500 ppm
I20260812 19:53:42.916661 32522 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:42.919281 32522 webserver.cc:533] Webserver started at http://127.25.14.1:36411/ using document root <none> and password file <none>
I20260812 19:53:42.920164 32522 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:42.920300 32522 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:42.927320 32522 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:53:42.930588 32546 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:42.932209 32522 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:53:42.932430 32522 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:42.933262 32522 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:42.961866 32522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:42.962842 32522 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:42.963160 32522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:42.964210 32522 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:42.966133 32522 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:42.966210 32522 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:42.966367 32522 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:42.966428 32522 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.002959 32522 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46519
I20260812 19:53:43.003011 32672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46519 every 8 connection(s)
I20260812 19:53:43.004684 32522 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:43.010028 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32522
I20260812 19:53:43.010279 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:53:43.015422 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:43.029486 32676 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:43.029567 32673 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:43.029981 32673 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.029987 32676 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.029520 32675 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:43.030864 32675 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.031282 32673 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:43.031309 32676 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:43.031769 32675 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:43.033706 32264 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:43.034034 32452 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:43.034787 32359 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:43.035903 32264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:37481
W20260812 19:53:43.391139 32685 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:43.391549 32685 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:43.391719 32685 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:43.401304 32685 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:43.401527 32685 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:53:43.412292 32685 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:43.414204 32685 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:43.416214 32685 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:43.427065 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:43.427100 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:43.427491 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:43.427927 32685 server_base.cc:1061] running on GCE node
I20260812 19:53:43.428611 32685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:43.430087 32685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:43.431381 32685 hybrid_clock.cc:648] HybridClock initialized: now 1786564423431218 us; error 125 us; skew 500 ppm
I20260812 19:53:43.431717 32685 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:43.434173 32685 webserver.cc:533] Webserver started at http://127.25.14.2:38247/ using document root <none> and password file <none>
I20260812 19:53:43.435019 32685 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:43.435144 32685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:43.441279 32685 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:53:43.444583 32699 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.446292 32685 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:53:43.446478 32685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "ccc31e8c1308402d96f39e5e8d8eb43e"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:43.447304 32685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:43.478121 32685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:43.479149 32685 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:43.479418 32685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:43.480401 32685 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:43.482414 32685 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:43.482496 32685 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.482596 32685 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:43.482640 32685 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.518304 32529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3ef649735d074b13845c3ee20ee3af82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:43.518615 32529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:43.520030 32685 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:45499
I20260812 19:53:43.520148 362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:45499 every 8 connection(s)
I20260812 19:53:43.521812 32685 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:53:43.529049 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32685
I20260812 19:53:43.529276 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:53:43.533331 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:43.544977 364 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:43.545495 364 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.546824 364 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:43.548054 363 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:43.548203 363 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.548729 363 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:43.549939 32452 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:45499)
I20260812 19:53:43.550098 32264 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:45499)
I20260812 19:53:43.551263 32264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:54045
I20260812 19:53:43.553368 365 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:43.553526 365 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:43.553915 365 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:43.555202 32359 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:45499)
W20260812 19:53:43.920112 373 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:43.920524 373 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:43.920673 373 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:43.930222 373 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:43.930485 373 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:53:43.941248 373 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:43.943120 373 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:43.945122 373 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:43.956117 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:43.956332 382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:43.956257 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:43.958253 373 server_base.cc:1061] running on GCE node
I20260812 19:53:43.958921 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:43.960158 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:43.961409 373 hybrid_clock.cc:648] HybridClock initialized: now 1786564423961256 us; error 113 us; skew 500 ppm
I20260812 19:53:43.961825 373 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:43.964169 373 webserver.cc:533] Webserver started at http://127.25.14.3:44357/ using document root <none> and password file <none>
I20260812 19:53:43.965052 373 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:43.965221 373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:43.971230 373 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 19:53:43.974162 390 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.975622 373 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:53:43.975816 373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "68476b355405493eaad1c0bba70a70f2"
format_stamp: "Formatted at 2026-08-12 19:53:36 on dist-test-slave-p3t5"
I20260812 19:53:43.976670 373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:43.991127 373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:43.992137 373 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:43.992455 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:43.993499 373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:43.995337 373 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:43.995445 373 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:43.995600 373 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:43.995677 373 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:44.032840 373 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:43977
I20260812 19:53:44.032877 519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:43977 every 8 connection(s)
I20260812 19:53:44.034648 373 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:53:44.038446 32675 heartbeater.cc:499] Master 127.25.14.62:45955 was elected leader, sending a full tablet report...
I20260812 19:53:44.040629 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 373
I20260812 19:53:44.040874 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:53:44.054654 521 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:44.055130 521 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:44.055779 523 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:44.055956 523 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:44.056430 521 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:44.056473 523 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:44.057549 522 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:44.057749 522 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:44.058326 522 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:44.058648 32359 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:43977)
I20260812 19:53:44.058761 32452 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:43977)
I20260812 19:53:44.059331 32264 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:43977)
I20260812 19:53:44.060251 32264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:59875
I20260812 19:53:44.068624 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:44.076315 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:53:44.076519 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32228
I20260812 19:53:44.077401 32297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:44.189642 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32228
W20260812 19:53:44.221151 32345 connection.cc:573] server connection from 127.0.0.1:59624 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:44.221251 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--webserver_interface=127.25.14.62
--webserver_port=34097
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381 with env {}
I20260812 19:53:44.553505 364 heartbeater.cc:499] Master 127.25.14.62:45955 was elected leader, sending a full tablet report...
W20260812 19:53:44.555336 364 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:45955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:45955: connect: Connection refused (error 111)
W20260812 19:53:44.598874 531 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:44.599251 531 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:44.599377 531 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:44.608249 531 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:53:44.608357 531 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:44.608445 531 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:53:44.608533 531 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:53:44.619305 531 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=34097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:44.621147 531 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:44.623292 531 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:44.632664 537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:44.632720 538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:44.632572 541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:44.633883 531 server_base.cc:1061] running on GCE node
I20260812 19:53:44.634850 531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:44.636739 531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:44.638144 531 hybrid_clock.cc:648] HybridClock initialized: now 1786564424638010 us; error 93 us; skew 500 ppm
I20260812 19:53:44.638540 531 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:44.641134 531 webserver.cc:533] Webserver started at http://127.25.14.62:34097/ using document root <none> and password file <none>
I20260812 19:53:44.642077 531 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:44.642225 531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:44.648072 531 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:53:44.651355 547 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:44.652850 531 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:53:44.653044 531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3ef649735d074b13845c3ee20ee3af82"
format_stamp: "Formatted at 2026-08-12 19:53:33 on dist-test-slave-p3t5"
I20260812 19:53:44.653803 531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:44.669521 531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:44.670514 531 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:44.670827 531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:44.693145 531 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:45955
I20260812 19:53:44.693181 604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:45955 every 8 connection(s)
I20260812 19:53:44.695032 531 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:53:44.697285 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 531
I20260812 19:53:44.697602 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32522
I20260812 19:53:44.704039 606 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:53:44.708777 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap starting.
I20260812 19:53:44.710770 32668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:44.728349 606 log.cc:826] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Log is configured to *not* fsync() on all Append() calls
I20260812 19:53:44.738850 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:53:44.739507 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: Bootstrap complete.
I20260812 19:53:44.747046 606 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:44.747887 606 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef649735d074b13845c3ee20ee3af82, State: Initialized, Role: FOLLOWER
I20260812 19:53:44.748777 606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:44.749862 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:44.750172 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:44.751111 606 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:53:44.762662 625 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:44.762903 625 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: loading cluster ID for follower catalog manager: success
I20260812 19:53:44.768493 625 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: acquiring CA information for follower catalog manager: success
I20260812 19:53:44.771919 625 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:53:44.817059 531 master_runner.cc:264] No leader master found from master 3ef649735d074b13845c3ee20ee3af82
I20260812 19:53:44.817279 531 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:53:44.824082 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32522
I20260812 19:53:44.850237 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46519
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=36411
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:45.062527 522 heartbeater.cc:499] Master 127.25.14.62:45955 was elected leader, sending a full tablet report...
I20260812 19:53:45.086751 566 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "68476b355405493eaad1c0bba70a70f2" instance_seqno: 1786564424024080) as {username='slave'} at 127.25.14.3:37483; Asking this server to re-register.
I20260812 19:53:45.087832 522 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:45.089886 566 ts_manager.cc:194] Registered new tserver with Master: 68476b355405493eaad1c0bba70a70f2 (127.25.14.3:43977)
W20260812 19:53:45.250798 631 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:45.251230 631 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:45.251372 631 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:45.260339 631 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:45.260583 631 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:45.271437 631 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46519
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36411
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:45.273294 631 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:45.275507 631 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:45.287034 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:45.287043 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:45.287837 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:45.288053 631 server_base.cc:1061] running on GCE node
I20260812 19:53:45.288710 631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:45.290055 631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:45.293480 631 hybrid_clock.cc:648] HybridClock initialized: now 1786564425293273 us; error 154 us; skew 500 ppm
I20260812 19:53:45.293958 631 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:45.296679 631 webserver.cc:533] Webserver started at http://127.25.14.1:36411/ using document root <none> and password file <none>
I20260812 19:53:45.297801 631 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:45.297966 631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:45.304327 631 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 19:53:45.307821 650 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:45.309544 631 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:53:45.309868 631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:45.310714 631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:45.338402 631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:45.339496 631 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:45.339861 631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:45.341005 631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:45.343099 631 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:45.343218 631 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:45.343384 631 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:45.343466 631 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:45.370257 729 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3ef649735d074b13845c3ee20ee3af82)
I20260812 19:53:45.370556 729 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:45.372797 729 leader_election.cc:290] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955), 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565)
I20260812 19:53:45.373906 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7" is_pre_election: true
I20260812 19:53:45.382432 577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3ef649735d074b13845c3ee20ee3af82" is_pre_election: true
I20260812 19:53:45.383216 577 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3fa8e711a5c47b2a10028be5d057ad3 in term 2.
I20260812 19:53:45.384155 32436 leader_election.cc:304] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3; no voters: 153a45b172f642818c4340f35e4d8ef7
I20260812 19:53:45.384796 729 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:53:45.384965 729 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3ef649735d074b13845c3ee20ee3af82)
I20260812 19:53:45.385110 729 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:53:45.388638 729 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:45.389518 729 leader_election.cc:290] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 3 election: Requested vote from peers 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955), 153a45b172f642818c4340f35e4d8ef7 (127.25.14.61:40565)
I20260812 19:53:45.390260 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "153a45b172f642818c4340f35e4d8ef7"
I20260812 19:53:45.390383 577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3ef649735d074b13845c3ee20ee3af82"
I20260812 19:53:45.390712 577 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:53:45.394870 631 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46519
I20260812 19:53:45.394932 771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46519 every 8 connection(s)
I20260812 19:53:45.397140 631 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:45.397814 577 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3fa8e711a5c47b2a10028be5d057ad3 in term 3.
I20260812 19:53:45.398738 32436 leader_election.cc:304] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3; no voters: 153a45b172f642818c4340f35e4d8ef7
I20260812 19:53:45.399427 729 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:53:45.399987 729 raft_consensus.cc:697] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [term 3 LEADER]: Becoming Leader. State: Replica: b3fa8e711a5c47b2a10028be5d057ad3, State: Running, Role: LEADER
I20260812 19:53:45.400642 729 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:45.403220 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 631
I20260812 19:53:45.403623 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 631
I20260812 19:53:45.409368 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.410789 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:45.413692 785 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:53:45.419003 785 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:53:45.420336 785 catalog_manager.cc:1295] Loaded cluster ID: 8ee4ffc4f1f54acab167da13333ee382
I20260812 19:53:45.420440 785 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:53:45.422307 785 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:53:45.423935 785 catalog_manager.cc:6103] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3: Loaded TSK: 0
I20260812 19:53:45.426152 785 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:53:45.432472 767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:45.522735 773 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:45.523628 773 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:45.524916 773 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:45.527160 772 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:45.527187 566 ts_manager.cc:194] Registered new tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:45.527338 772 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:45.528123 772 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:45.528568 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 631
I20260812 19:53:45.530912 32452 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:45.532828 32452 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:46631
W20260812 19:53:45.551506 32342 connection.cc:573] server connection from 127.25.14.1:35491 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 19:53:45.551548 32437 connection.cc:573] server connection from 127.25.14.1:46631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:53:45.552670 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46519
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=36411
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260812 19:53:45.563510 364 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:45.563764 364 heartbeater.cc:499] Master 127.25.14.62:45955 was elected leader, sending a full tablet report...
I20260812 19:53:45.564764 566 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ccc31e8c1308402d96f39e5e8d8eb43e" instance_seqno: 1786564423510594) as {username='slave'} at 127.25.14.2:43833; Asking this server to re-register.
I20260812 19:53:45.565433 364 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:45.566627 566 ts_manager.cc:194] Registered new tserver with Master: ccc31e8c1308402d96f39e5e8d8eb43e (127.25.14.2:45499)
I20260812 19:53:45.691763 793 raft_consensus.cc:493] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3ef649735d074b13845c3ee20ee3af82)
I20260812 19:53:45.691989 793 raft_consensus.cc:515] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } }
I20260812 19:53:45.694115 793 leader_election.cc:290] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ef649735d074b13845c3ee20ee3af82 (127.25.14.62:45955), b3fa8e711a5c47b2a10028be5d057ad3 (127.25.14.60:44381)
I20260812 19:53:45.718794 577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153a45b172f642818c4340f35e4d8ef7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3ef649735d074b13845c3ee20ee3af82" is_pre_election: true
I20260812 19:53:45.719295 577 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153a45b172f642818c4340f35e4d8ef7 in current term 3: Already voted for candidate b3fa8e711a5c47b2a10028be5d057ad3 in this term.
I20260812 19:53:45.720767 32465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153a45b172f642818c4340f35e4d8ef7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" is_pre_election: true
I20260812 19:53:45.721532 32346 leader_election.cc:304] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 153a45b172f642818c4340f35e4d8ef7; no voters: 3ef649735d074b13845c3ee20ee3af82, b3fa8e711a5c47b2a10028be5d057ad3
I20260812 19:53:45.722043 793 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:53:45.724666 793 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:53:45.882923 32372 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [term 3 FOLLOWER]: Refusing update from remote peer b3fa8e711a5c47b2a10028be5d057ad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:53:45.883917 775 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:53:45.886173 793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.886708 793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:45.889482 793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.889892 793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153a45b172f642818c4340f35e4d8ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:45.893666 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.893944 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
I20260812 19:53:45.894510 577 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [term 3 FOLLOWER]: Refusing update from remote peer b3fa8e711a5c47b2a10028be5d057ad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:53:45.895437 800 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:53:45.899737 804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.900103 804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:53:45.900485 801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fa8e711a5c47b2a10028be5d057ad3. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:45.900722 801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef649735d074b13845c3ee20ee3af82 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:53:45.957433 791 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:45.957880 791 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:45.958036 791 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:53:45.958135 791 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:45.967026 791 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:45.967283 791 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:45.977880 791 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46519
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36411
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:45.979728 791 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:45.981943 791 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:45.992719 813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:45.992798 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:45.992753 815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:45.994161 791 server_base.cc:1061] running on GCE node
I20260812 19:53:45.994808 791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:45.996014 791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:45.997347 791 hybrid_clock.cc:648] HybridClock initialized: now 1786564425997182 us; error 114 us; skew 500 ppm
I20260812 19:53:45.997717 791 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:46.000027 791 webserver.cc:533] Webserver started at http://127.25.14.1:36411/ using document root <none> and password file <none>
I20260812 19:53:46.000846 791 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:46.001012 791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:46.007648 791 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:53:46.010968 821 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:46.012580 791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:53:46.012797 791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:46.013589 791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:46.033876 791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:46.035029 791 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:46.035328 791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:46.036551 791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:46.038424 791 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:46.038522 791 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:46.038597 791 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:46.038663 791 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:53:46.038990 791 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:53:46.039058 791 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:53:46.039176 791 fault_injection.cc:59] injecting fault for kudu (pid 791): FLAGS_fault_before_start (process will exit)
I20260812 19:53:46.063366 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46519
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--webserver_port=36411
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:53:46.358038 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef649735d074b13845c3ee20ee3af82" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45955 } } peers { permanent_uuid: "153a45b172f642818c4340f35e4d8ef7" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40565 } } peers { permanent_uuid: "b3fa8e711a5c47b2a10028be5d057ad3" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44381 } } }
I20260812 19:53:46.358353 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fa8e711a5c47b2a10028be5d057ad3 [sys.catalog]: This master's current role is: LEADER
W20260812 19:53:46.440178 831 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:53:46.440519 831 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:53:46.440701 831 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:53:46.449690 831 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:53:46.449962 831 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:53:46.460641 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42461,127.25.14.21:39057,127.25.14.22:44925,127.25.14.23:33725,127.25.14.24:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46519
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36411
--tserver_master_addrs=127.25.14.62:45955,127.25.14.61:40565,127.25.14.60:44381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:53:46.462528 831 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:53:46.464462 831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:53:46.475447 839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:46.475476 840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:53:46.475615 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:53:46.476279 831 server_base.cc:1061] running on GCE node
I20260812 19:53:46.477108 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:53:46.478515 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:53:46.479857 831 hybrid_clock.cc:648] HybridClock initialized: now 1786564426479684 us; error 108 us; skew 500 ppm
I20260812 19:53:46.480214 831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:53:46.482714 831 webserver.cc:533] Webserver started at http://127.25.14.1:36411/ using document root <none> and password file <none>
I20260812 19:53:46.483625 831 fs_manager.cc:362] Metadata directory not provided
I20260812 19:53:46.483772 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:53:46.489591 831 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:53:46.492630 848 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:46.494159 831 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:53:46.494364 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "ea98f130a1d64097b4c9e959e8f0f402"
format_stamp: "Formatted at 2026-08-12 19:53:35 on dist-test-slave-p3t5"
I20260812 19:53:46.495096 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:53:46.520341 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:53:46.521368 831 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:53:46.521790 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:53:46.522871 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:53:46.524935 831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:53:46.525041 831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:46.525206 831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:53:46.525287 831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:53:46.555716 363 heartbeater.cc:499] Master 127.25.14.60:44381 was elected leader, sending a full tablet report...
I20260812 19:53:46.566920 831 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46519
I20260812 19:53:46.566980 977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46519 every 8 connection(s)
I20260812 19:53:46.569002 831 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:53:46.570987 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 831
I20260812 19:53:46.571363 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 831
I20260812 19:53:46.591928 978 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44381
I20260812 19:53:46.594259 978 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:46.595558 978 heartbeater.cc:507] Master 127.25.14.60:44381 requested a full tablet report, sending...
I20260812 19:53:46.597525 980 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40565
I20260812 19:53:46.597689 980 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:46.597648 32452 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:46.598229 980 heartbeater.cc:507] Master 127.25.14.61:40565 requested a full tablet report, sending...
I20260812 19:53:46.598834 32452 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:47637
I20260812 19:53:46.599686 32359 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:46.606781 979 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45955
I20260812 19:53:46.606995 979 heartbeater.cc:461] Registering TS with master...
I20260812 19:53:46.607421 979 heartbeater.cc:507] Master 127.25.14.62:45955 requested a full tablet report, sending...
I20260812 19:53:46.608924 566 ts_manager.cc:194] Re-registered known tserver with Master: ea98f130a1d64097b4c9e959e8f0f402 (127.25.14.1:46519)
I20260812 19:53:46.609692 972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:46.705101 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 831
I20260812 19:53:46.733289 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32685
I20260812 19:53:46.742336 355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:46.831844 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32685
I20260812 19:53:46.854709 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 373
I20260812 19:53:46.864107 514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:46.954850 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 373
I20260812 19:53:46.980089 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 531
I20260812 19:53:46.986807 598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:47.097033 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 531
I20260812 19:53:47.125581 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32305
I20260812 19:53:47.126965 32392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:47.237833 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32305
I20260812 19:53:47.265120 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32400
I20260812 19:53:47.266505 32487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:53:47.378752 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32400
2026-08-12T19:53:47Z chronyd exiting
2026-08-12T19:53:47Z chronyd exiting
2026-08-12T19:53:47Z chronyd exiting
2026-08-12T19:53:47Z chronyd exiting
2026-08-12T19:53:47Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/2 (14866 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:53:55: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:54:04.345162 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
W20260812 19:54:04.725312 1319 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:04.725711 1319 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:04.725899 1319 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:04.735013 1319 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:04.735111 1319 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:04.735208 1319 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:04.735306 1319 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:04.745752 1319 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:04.747674 1319 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:04.749732 1319 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:04.759153 1327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:04.759160 1324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:04.759435 1325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:04.760320 1319 server_base.cc:1061] running on GCE node
I20260812 19:54:04.761230 1319 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:04.761430 1319 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:04.761540 1319 hybrid_clock.cc:648] HybridClock initialized: now 1786564444761497 us; error 0 us; skew 500 ppm
I20260812 19:54:04.761960 1319 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:04.765110 1319 webserver.cc:533] Webserver started at http://127.25.14.62:35327/ using document root <none> and password file <none>
I20260812 19:54:04.766150 1319 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:04.766269 1319 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:04.766670 1319 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:04.769508 1319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:04.770568 1319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:04.777122 1319 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:54:04.781443 1332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:04.783210 1319 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:54:04.783468 1319 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:04.783761 1319 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:04.818691 1319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:04.819818 1319 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:04.820161 1319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:04.843791 1319 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40607
I20260812 19:54:04.843847 1389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40607 every 8 connection(s)
I20260812 19:54:04.845709 1319 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:04.850903 1391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:04.854970 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1319
I20260812 19:54:04.855222 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:54:04.856632 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46307
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
I20260812 19:54:04.859022 1391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:04.885174 1391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:04.886619 1333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
W20260812 19:54:04.888485 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
I20260812 19:54:04.937914 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } attempt: 1
W20260812 19:54:04.940327 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
I20260812 19:54:05.008737 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } attempt: 2
W20260812 19:54:05.011812 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
I20260812 19:54:05.103193 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } attempt: 3
W20260812 19:54:05.105680 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
W20260812 19:54:05.240674 1396 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:05.241031 1396 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:05.241101 1396 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:54:05.248998 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } attempt: 4
W20260812 19:54:05.250101 1396 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:05.250223 1396 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:05.250327 1396 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:05.250407 1396 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 19:54:05.251497 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111)
I20260812 19:54:05.261338 1396 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:05.263195 1396 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:05.265206 1396 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:05.274747 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:05.274699 1404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:05.275586 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:05.276262 1396 server_base.cc:1061] running on GCE node
I20260812 19:54:05.277158 1396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:05.277333 1396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:05.277411 1396 hybrid_clock.cc:648] HybridClock initialized: now 1786564445277391 us; error 0 us; skew 500 ppm
I20260812 19:54:05.277810 1396 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:05.281015 1396 webserver.cc:533] Webserver started at http://127.25.14.61:33307/ using document root <none> and password file <none>
I20260812 19:54:05.281931 1396 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:05.282038 1396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:05.282375 1396 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:05.284866 1396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "ee49622bf94e4b82bdd6a59bf585b581"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.285852 1396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "ee49622bf94e4b82bdd6a59bf585b581"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.291919 1396 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:54:05.295771 1413 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:05.297335 1396 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:54:05.297546 1396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "ee49622bf94e4b82bdd6a59bf585b581"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.297871 1396 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:05.320670 1396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:05.321825 1396 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:05.322172 1396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:05.345533 1396 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46307
I20260812 19:54:05.345554 1472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46307 every 8 connection(s)
I20260812 19:54:05.347335 1396 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:54:05.347936 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1396
I20260812 19:54:05.348140 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:54:05.349422 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37367
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
I20260812 19:54:05.352422 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:05.359139 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:05.375370 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:05.385984 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:05.387281 1415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
W20260812 19:54:05.389276 1473 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.438733 1473 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 1
W20260812 19:54:05.441213 1473 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.509533 1473 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 2
W20260812 19:54:05.512156 1473 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.550814 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } attempt: 5
I20260812 19:54:05.558076 1391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:05.560230 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.603477 1473 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 3
W20260812 19:54:05.606007 1473 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.611522 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 1
W20260812 19:54:05.613737 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.681983 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 2
W20260812 19:54:05.684368 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
W20260812 19:54:05.730152 1476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:05.730521 1476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:05.730602 1476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:05.739387 1476 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:05.739490 1476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:05.739548 1476 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:05.739590 1476 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:05.749303 1473 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 4
I20260812 19:54:05.751044 1476 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
W20260812 19:54:05.751603 1473 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.753011 1476 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:05.755009 1476 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:05.764381 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:05.764413 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:05.764835 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:05.766059 1476 server_base.cc:1061] running on GCE node
I20260812 19:54:05.766934 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:05.767117 1476 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:05.767237 1476 hybrid_clock.cc:648] HybridClock initialized: now 1786564445767213 us; error 0 us; skew 500 ppm
I20260812 19:54:05.767613 1476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:05.770588 1476 webserver.cc:533] Webserver started at http://127.25.14.60:37703/ using document root <none> and password file <none>
I20260812 19:54:05.771416 1476 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:05.771544 1476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:05.771945 1476 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:05.777055 1476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.778039 1476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.784451 1476 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260812 19:54:05.788725 1493 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:05.790557 1476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:54:05.790680 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 3
I20260812 19:54:05.790813 1476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:05.791131 1476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 19:54:05.792919 1391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111)
I20260812 19:54:05.814682 1476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:05.815788 1476 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:05.816114 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:05.838620 1476 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:37367
I20260812 19:54:05.838662 1550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:37367 every 8 connection(s)
I20260812 19:54:05.840459 1476 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:54:05.845409 1552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:05.848436 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1476
I20260812 19:54:05.848680 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:54:05.854463 1552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:05.869174 1552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:05.876386 1552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:05.888151 1552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Bootstrap starting.
I20260812 19:54:05.892549 1552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:05.893921 1552 log.cc:826] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:05.896806 1552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: No bootstrap required, opened a new log
I20260812 19:54:05.902855 1552 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:05.903231 1552 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:05.903388 1552 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6073ff6adbcd4bc9950b1c1c38baa0a2, State: Initialized, Role: FOLLOWER
I20260812 19:54:05.904112 1552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:05.905267 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:05.905786 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:05.906751 1552 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:05.912050 1572 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 19:54:05.915809 1476 master_runner.cc:187] Error getting master registration for 127.25.14.62:40607: OK, instance_id { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" instance_seqno: 1786564444837571 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:54:05.920461 1476 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:05.935458 1561 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:54:05.935811 1561 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:05.937976 1561 leader_election.cc:290] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607), ee49622bf94e4b82bdd6a59bf585b581 (127.25.14.61:46307)
I20260812 19:54:05.970149 1391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 4
I20260812 19:54:05.983636 1391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap starting.
I20260812 19:54:05.987449 1391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:05.988727 1391 log.cc:826] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:05.992230 1391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: No bootstrap required, opened a new log
I20260812 19:54:06.000684 1391 raft_consensus.cc:359] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.001116 1391 raft_consensus.cc:385] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:06.001258 1391 raft_consensus.cc:740] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841289fe039443ffb918fd0de05b5d05, State: Initialized, Role: FOLLOWER
I20260812 19:54:06.002130 1391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.003335 1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.003691 1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.004623 1391 sys_catalog.cc:565] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:06.004732 1363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841289fe039443ffb918fd0de05b5d05" is_pre_election: true
I20260812 19:54:06.005327 1363 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6073ff6adbcd4bc9950b1c1c38baa0a2 in term 0.
I20260812 19:54:06.006151 1495 leader_election.cc:304] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6073ff6adbcd4bc9950b1c1c38baa0a2, 841289fe039443ffb918fd0de05b5d05; no voters:
I20260812 19:54:06.006698 1561 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:54:06.006927 1561 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:54:06.007088 1561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:54:06.011157 1561 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.012180 1561 leader_election.cc:290] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 1 election: Requested vote from peers 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607), ee49622bf94e4b82bdd6a59bf585b581 (127.25.14.61:46307)
I20260812 19:54:06.012737 1363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841289fe039443ffb918fd0de05b5d05"
I20260812 19:54:06.012980 1363 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:54:06.016927 1589 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:54:06.017062 1589 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:54:06.017257 1363 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6073ff6adbcd4bc9950b1c1c38baa0a2 in term 1.
I20260812 19:54:06.018730 1495 leader_election.cc:304] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6073ff6adbcd4bc9950b1c1c38baa0a2, 841289fe039443ffb918fd0de05b5d05; no voters:
I20260812 19:54:06.019755 1561 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:54:06.020470 1561 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6073ff6adbcd4bc9950b1c1c38baa0a2, State: Running, Role: LEADER
I20260812 19:54:06.021096 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.025246 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6073ff6adbcd4bc9950b1c1c38baa0a2. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.025640 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:06.023056 1319 master_runner.cc:187] Error getting master registration for 127.25.14.61:46307: OK, instance_id { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" instance_seqno: 1786564445339493 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:54:06.031968 1319 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:06.032748 1593 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:54:06.038928 1593 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:54:06.050832 1473 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } attempt: 5
I20260812 19:54:06.058954 1363 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Refusing update from remote peer 6073ff6adbcd4bc9950b1c1c38baa0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:54:06.060151 1591 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:54:06.064291 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Bootstrap starting.
W20260812 19:54:06.068598 1499 tablet.cc:2406] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 19:54:06.073407 1339 tablet.cc:2406] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:54:06.075939 1473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:06.078958 1473 log.cc:826] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:06.085899 1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6073ff6adbcd4bc9950b1c1c38baa0a2. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.086222 1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.087509 1595 mvcc.cc:204] Tried to move back new op lower bound from 7317767971035062272 to 7317767970908561408. Current Snapshot: MvccSnapshot[applied={T|T < 7317767971035062272}]
I20260812 19:54:06.090727 1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.090975 1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.093864 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: No bootstrap required, opened a new log
I20260812 19:54:06.105258 1473 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.105767 1473 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:06.105931 1473 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee49622bf94e4b82bdd6a59bf585b581, State: Initialized, Role: FOLLOWER
I20260812 19:54:06.106163 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.106412 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:06.106786 1473 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:06.109594 1473 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:06.109591 1593 catalog_manager.cc:1383] Generated new cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:06.109783 1593 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:54:06.115119 1441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee49622bf94e4b82bdd6a59bf585b581"
I20260812 19:54:06.115667 1441 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:54:06.117704 1443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee49622bf94e4b82bdd6a59bf585b581" is_pre_election: true
I20260812 19:54:06.122723 1441 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6073ff6adbcd4bc9950b1c1c38baa0a2 in term 1.
I20260812 19:54:06.126859 1440 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Refusing update from remote peer 6073ff6adbcd4bc9950b1c1c38baa0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:54:06.129144 1561 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:54:06.108798 1603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.144210 1603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.150278 1593 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:54:06.152355 1593 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:54:06.178524 1606 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6073ff6adbcd4bc9950b1c1c38baa0a2. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.178855 1606 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.179199 1603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.179378 1603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:06.180706 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:06.180948 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:06.187628 1593 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Generated new TSK 0
I20260812 19:54:06.189235 1593 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:54:06.196208 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
I20260812 19:54:06.234752 1622 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:06.235008 1622 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: loading cluster ID for follower catalog manager: success
I20260812 19:54:06.239427 1622 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: acquiring CA information for follower catalog manager: success
I20260812 19:54:06.242412 1622 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:54:06.777783 1623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:06.778266 1623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:06.778393 1623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:06.792701 1623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:06.792982 1623 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:06.810434 1623 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:06.812741 1623 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:06.815182 1623 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:06.827108 1631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:06.827808 1632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:06.836225 1623 server_base.cc:1061] running on GCE node
W20260812 19:54:06.836536 1635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:06.837325 1623 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:06.837594 1623 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:06.837733 1623 hybrid_clock.cc:648] HybridClock initialized: now 1786564446837704 us; error 0 us; skew 500 ppm
I20260812 19:54:06.838198 1623 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:06.841343 1623 webserver.cc:533] Webserver started at http://127.25.14.1:39377/ using document root <none> and password file <none>
I20260812 19:54:06.842442 1623 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:06.842586 1623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:06.843123 1623 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:06.846702 1623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:06.847867 1623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:06.855892 1623 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.007s
I20260812 19:54:06.860819 1639 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:06.862690 1623 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:54:06.862895 1623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:06.863158 1623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:06.886021 1623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:06.887032 1623 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:06.887297 1623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:06.888331 1623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:06.890494 1623 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:06.890588 1623 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:06.890669 1623 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:06.890731 1623 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:06.929934 1623 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45529
I20260812 19:54:06.930051 1770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45529 every 8 connection(s)
I20260812 19:54:06.931563 1623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:06.935298 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1623
I20260812 19:54:06.935525 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:54:06.942414 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
I20260812 19:54:06.955053 1773 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:06.955581 1773 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:06.957089 1773 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:06.960952 1431 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:06.974534 1772 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:06.974766 1772 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:06.975440 1772 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:06.977751 1771 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:06.977921 1771 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:06.977885 1350 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:06.978472 1771 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:06.980206 1511 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:06.981928 1511 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:59707
I20260812 19:54:07.019832 1589 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:07.019969 1589 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: loading cluster ID for follower catalog manager: success
I20260812 19:54:07.024175 1589 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: acquiring CA information for follower catalog manager: success
I20260812 19:54:07.027122 1589 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:54:07.339805 1783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:07.340165 1783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:07.340242 1783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:07.349068 1783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:07.349248 1783 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:54:07.359925 1783 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:07.361763 1783 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:07.363669 1783 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:07.374503 1789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:07.374470 1788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:07.375417 1791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:07.375680 1783 server_base.cc:1061] running on GCE node
I20260812 19:54:07.376364 1783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:07.376570 1783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:07.376681 1783 hybrid_clock.cc:648] HybridClock initialized: now 1786564447376655 us; error 0 us; skew 500 ppm
I20260812 19:54:07.377055 1783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:07.379776 1783 webserver.cc:533] Webserver started at http://127.25.14.2:41777/ using document root <none> and password file <none>
I20260812 19:54:07.380692 1783 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:07.380864 1783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:07.381279 1783 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:07.384039 1783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "d64ce738a6874f12b9543a4063d3bb95"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.384945 1783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "d64ce738a6874f12b9543a4063d3bb95"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.391204 1783 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 19:54:07.395097 1798 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:07.396615 1783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260812 19:54:07.396827 1783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "d64ce738a6874f12b9543a4063d3bb95"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.397076 1783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:07.428125 1783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:07.429211 1783 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:07.429540 1783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:07.430737 1783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:07.432682 1783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:07.432833 1783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:07.432978 1783 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:07.433071 1783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:07.471639 1783 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:42859
I20260812 19:54:07.471710 1923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:42859 every 8 connection(s)
I20260812 19:54:07.473256 1783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:54:07.480216 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1783
I20260812 19:54:07.480517 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:54:07.491117 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
I20260812 19:54:07.504379 1926 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:07.505031 1926 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:07.506316 1926 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:07.506685 1925 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:07.506878 1925 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:07.507900 1925 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:07.508497 1431 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:42859)
I20260812 19:54:07.509538 1350 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:42859)
I20260812 19:54:07.512092 1924 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:07.512315 1924 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:07.512904 1924 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:07.514078 1511 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:42859)
I20260812 19:54:07.514959 1511 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:54763
W20260812 19:54:07.909102 1936 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:07.909514 1936 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:07.909689 1936 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:07.919255 1936 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:07.919512 1936 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:54:07.930205 1936 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:07.932035 1936 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:07.934207 1936 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:07.945495 1942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:07.945516 1943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:07.946319 1945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:07.947705 1936 server_base.cc:1061] running on GCE node
I20260812 19:54:07.948395 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:07.948590 1936 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:07.948716 1936 hybrid_clock.cc:648] HybridClock initialized: now 1786564447948664 us; error 0 us; skew 500 ppm
I20260812 19:54:07.949100 1936 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:07.951690 1936 webserver.cc:533] Webserver started at http://127.25.14.3:42435/ using document root <none> and password file <none>
I20260812 19:54:07.952615 1936 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:07.952754 1936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:07.953171 1936 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:07.955983 1936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "c2d51bd329c04985956c7593847cbc63"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.956924 1936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "c2d51bd329c04985956c7593847cbc63"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.963272 1936 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 19:54:07.967669 1950 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:07.969342 1936 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:54:07.969569 1936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "c2d51bd329c04985956c7593847cbc63"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:07.969875 1936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:07.984663 1771 heartbeater.cc:499] Master 127.25.14.60:37367 was elected leader, sending a full tablet report...
I20260812 19:54:07.987511 1936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:07.988538 1936 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:07.988849 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:07.989987 1936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:07.991891 1936 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:07.991987 1936 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:07.992149 1936 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:07.992245 1936 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:08.032465 1936 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:36181
I20260812 19:54:08.032511 2081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:36181 every 8 connection(s)
I20260812 19:54:08.034112 1936 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:54:08.038337 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1936
I20260812 19:54:08.038550 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:54:08.056002 2082 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:08.056198 2083 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:08.056241 2084 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:08.056565 2084 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:08.056560 2082 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:08.056564 2083 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:08.057899 2084 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:08.057931 2082 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:08.057996 2083 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:08.059980 1431 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:36181)
I20260812 19:54:08.059959 1511 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:36181)
I20260812 19:54:08.060305 1350 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:36181)
I20260812 19:54:08.061203 1511 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:46999
I20260812 19:54:08.072254 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:08.080941 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:08.081149 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1319
I20260812 19:54:08.082178 1385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:08.193076 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1319
I20260812 19:54:08.224323 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--webserver_interface=127.25.14.62
--webserver_port=35327
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
W20260812 19:54:08.309020 1495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:40607: connect: Connection refused (error 111)
W20260812 19:54:08.310922 1495 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 -> Peer 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607): Couldn't send request to peer 841289fe039443ffb918fd0de05b5d05. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:40607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:54:08.513255 1925 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:40607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:40607: connect: Connection refused (error 111)
I20260812 19:54:08.517697 1924 heartbeater.cc:499] Master 127.25.14.60:37367 was elected leader, sending a full tablet report...
W20260812 19:54:08.602499 2093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:08.603047 2093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:08.603180 2093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:08.612106 2093 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:08.612205 2093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:08.612308 2093 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:08.612396 2093 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:08.622709 2093 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35327
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:08.624553 2093 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:08.626574 2093 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:08.636087 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:08.636196 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:08.636155 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:08.637348 2093 server_base.cc:1061] running on GCE node
I20260812 19:54:08.638255 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:08.638417 2093 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:08.638512 2093 hybrid_clock.cc:648] HybridClock initialized: now 1786564448638490 us; error 0 us; skew 500 ppm
I20260812 19:54:08.638823 2093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:08.641870 2093 webserver.cc:533] Webserver started at http://127.25.14.62:35327/ using document root <none> and password file <none>
I20260812 19:54:08.642691 2093 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:08.642812 2093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:08.648785 2093 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 19:54:08.651918 2111 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:08.653355 2093 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:54:08.653571 2093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:08.654354 2093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:08.668079 2093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:08.669075 2093 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:08.669493 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:08.692410 2093 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40607
I20260812 19:54:08.692430 2175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40607 every 8 connection(s)
I20260812 19:54:08.694428 2093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:08.702862 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2093
I20260812 19:54:08.702896 2176 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:08.703212 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1623
I20260812 19:54:08.708626 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap starting.
I20260812 19:54:08.714567 1766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:08.730394 2176 log.cc:826] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:08.740592 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:08.741243 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap complete.
I20260812 19:54:08.748739 2176 raft_consensus.cc:359] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:08.749533 2176 raft_consensus.cc:740] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841289fe039443ffb918fd0de05b5d05, State: Initialized, Role: FOLLOWER
I20260812 19:54:08.750414 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:08.751416 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:08.751754 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:08.752686 2176 sys_catalog.cc:565] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:08.765513 2194 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:08.765671 2194 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: loading cluster ID for follower catalog manager: success
I20260812 19:54:08.771629 2194 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: acquiring CA information for follower catalog manager: success
I20260812 19:54:08.778317 2194 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:08.820251 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1623
I20260812 19:54:08.821787 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:08.822113 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:08.872720 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6073ff6adbcd4bc9950b1c1c38baa0a2. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:08.873086 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:08.874938 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45529
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=39377
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
I20260812 19:54:09.063716 2082 heartbeater.cc:499] Master 127.25.14.60:37367 was elected leader, sending a full tablet report...
I20260812 19:54:09.073390 2129 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c2d51bd329c04985956c7593847cbc63" instance_seqno: 1786564448021856) as {username='slave'} at 127.25.14.3:53743; Asking this server to re-register.
I20260812 19:54:09.074678 2083 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:09.076464 2129 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:36181)
I20260812 19:54:09.198303 2201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:09.198639 2201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: LEADER
W20260812 19:54:09.283226 2207 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:09.283607 2207 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:09.283780 2207 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:09.292804 2207 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:09.293049 2207 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:09.304056 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39377
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:09.306017 2207 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:09.308439 2207 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:09.320307 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:09.320284 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:09.320655 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:09.321609 2207 server_base.cc:1061] running on GCE node
I20260812 19:54:09.322393 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:09.322609 2207 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:09.322722 2207 hybrid_clock.cc:648] HybridClock initialized: now 1786564449322697 us; error 0 us; skew 500 ppm
I20260812 19:54:09.323103 2207 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:09.330812 2207 webserver.cc:533] Webserver started at http://127.25.14.1:39377/ using document root <none> and password file <none>
I20260812 19:54:09.331773 2207 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:09.331975 2207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:09.338403 2207 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:09.341477 2222 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:09.343019 2207 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:09.343226 2207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:09.344039 2207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:09.360033 2207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:09.361654 2207 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:09.362025 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:09.363157 2207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:09.365036 2207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:09.365155 2207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:09.365319 2207 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:09.365394 2207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:09.404527 2207 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45529
I20260812 19:54:09.404633 2351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45529 every 8 connection(s)
I20260812 19:54:09.406476 2207 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:09.415208 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2207
I20260812 19:54:09.428370 2354 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:09.428836 2354 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:09.430294 2354 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:09.432001 2353 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:09.432168 2353 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:09.432757 2353 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:09.432919 1431 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:09.434093 2129 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:09.434434 2352 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:09.434587 2352 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:09.435148 2352 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:09.436504 1511 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:09.437378 1511 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:54007
I20260812 19:54:09.462469 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2207
I20260812 19:54:09.479547 2346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:09.521374 1925 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:09.522516 2129 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d64ce738a6874f12b9543a4063d3bb95" instance_seqno: 1786564447462381) as {username='slave'} at 127.25.14.2:46589; Asking this server to re-register.
I20260812 19:54:09.523147 1925 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:09.524369 2129 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:42859)
I20260812 19:54:09.596324 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2207
I20260812 19:54:09.628675 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45529
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=39377
--webserver_interface=127.25.14.1
--time_source=system_unsync
--fault_before_start=1.0 with env {}
W20260812 19:54:10.013989 2367 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:10.014360 2367 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:10.014451 2367 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:54:10.014516 2367 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:10.023533 2367 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:10.023746 2367 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:10.034346 2367 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39377
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:10.036963 2367 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:10.039490 2367 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:10.050338 2375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:10.050307 2374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:10.051198 2377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:10.051921 2367 server_base.cc:1061] running on GCE node
I20260812 19:54:10.052567 2367 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:10.052747 2367 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:10.052838 2367 hybrid_clock.cc:648] HybridClock initialized: now 1786564450052804 us; error 0 us; skew 500 ppm
I20260812 19:54:10.053120 2367 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:10.055751 2367 webserver.cc:533] Webserver started at http://127.25.14.1:39377/ using document root <none> and password file <none>
I20260812 19:54:10.056731 2367 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:10.056856 2367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:10.063333 2367 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:54:10.066841 2383 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:10.068820 2367 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:54:10.069051 2367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:10.069996 2367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:10.093214 2367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:10.094484 2367 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:10.094808 2367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:10.095993 2367 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:10.098027 2367 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:10.098161 2367 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:10.098294 2367 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:10.098387 2367 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:54:10.098791 2367 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:54:10.098915 2367 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:54:10.099014 2367 fault_injection.cc:59] injecting fault for kudu (pid 2367): FLAGS_fault_before_start (process will exit)
I20260812 19:54:10.122177 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45529
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=39377
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
W20260812 19:54:10.495322 2391 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:10.495707 2391 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:10.495870 2391 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:10.504832 2391 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:10.505087 2391 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:10.516405 2391 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39377
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:10.518400 2391 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:10.520468 2391 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:10.531850 2400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:10.531812 2398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:10.532292 2402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:10.533672 2391 server_base.cc:1061] running on GCE node
I20260812 19:54:10.534403 2391 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:10.534607 2391 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:10.534725 2391 hybrid_clock.cc:648] HybridClock initialized: now 1786564450534681 us; error 0 us; skew 500 ppm
I20260812 19:54:10.535171 2391 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:10.538107 2391 webserver.cc:533] Webserver started at http://127.25.14.1:39377/ using document root <none> and password file <none>
I20260812 19:54:10.539038 2391 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:10.539149 2391 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:10.545698 2391 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:54:10.548918 2407 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:10.550688 2391 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 19:54:10.550915 2391 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:10.551705 2391 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:10.588351 2391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:10.589349 2391 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:10.589679 2391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:10.590776 2391 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:10.592603 2391 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:10.592680 2391 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:10.592767 2391 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:10.592803 2391 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:10.631299 2391 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45529
I20260812 19:54:10.631363 2532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45529 every 8 connection(s)
I20260812 19:54:10.633003 2391 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:10.640300 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2391
I20260812 19:54:10.640769 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2391
I20260812 19:54:10.666623 2535 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:10.667191 2535 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:10.667433 2533 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:10.667572 2533 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:10.668637 2533 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:10.669075 2535 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:10.669481 2528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:10.670746 1431 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:10.670737 1511 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:45529)
I20260812 19:54:10.672660 1511 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:55789
I20260812 19:54:10.763669 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2391
I20260812 19:54:10.788290 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1783
I20260812 19:54:10.797544 1916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:10.885656 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1783
I20260812 19:54:10.910259 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1936
I20260812 19:54:10.919559 2076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:11.010076 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1936
I20260812 19:54:11.034626 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2093
I20260812 19:54:11.042799 2170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:11.149982 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2093
I20260812 19:54:11.177721 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1396
I20260812 19:54:11.179214 1468 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 19:54:11.191109 1495 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 -> Peer 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607): Couldn't send request to peer 841289fe039443ffb918fd0de05b5d05. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:40607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:54:11.291905 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1396
I20260812 19:54:11.323331 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1476
I20260812 19:54:11.324720 1546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:11.436491 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1476
Shutting down hive metastore.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:54:12: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:54:22.191211 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
W20260812 19:54:22.562911 2770 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:22.563441 2770 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:22.563585 2770 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:22.572799 2770 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:22.572909 2770 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:22.572969 2770 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:22.573010 2770 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:22.583698 2770 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:22.585507 2770 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:22.587668 2770 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:22.597601 2776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:22.597461 2777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:22.597986 2779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:22.598383 2770 server_base.cc:1061] running on GCE node
I20260812 19:54:22.599472 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:22.599675 2770 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:22.599774 2770 hybrid_clock.cc:648] HybridClock initialized: now 1786564462599754 us; error 0 us; skew 500 ppm
I20260812 19:54:22.600158 2770 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:22.603442 2770 webserver.cc:533] Webserver started at http://127.25.14.62:32909/ using document root <none> and password file <none>
I20260812 19:54:22.604279 2770 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:22.604380 2770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:22.610541 2770 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:22.613967 2787 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:22.615593 2770 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:54:22.615801 2770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:22.616607 2770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:22.643039 2770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:22.644104 2770 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:22.644438 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:22.666981 2770 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40607
I20260812 19:54:22.667007 2848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40607 every 8 connection(s)
I20260812 19:54:22.668860 2770 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:22.670642 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2770
I20260812 19:54:22.670850 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:54:22.672135 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46307
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
I20260812 19:54:22.679704 2849 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:22.684118 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap starting.
I20260812 19:54:22.702832 2849 log.cc:826] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:22.713088 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:22.713837 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap complete.
I20260812 19:54:22.720992 2849 raft_consensus.cc:359] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:22.721969 2849 raft_consensus.cc:740] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841289fe039443ffb918fd0de05b5d05, State: Initialized, Role: FOLLOWER
I20260812 19:54:22.722824 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:22.723785 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:22.724140 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:22.724752 2849 sys_catalog.cc:565] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:22.735888 2864 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:22.736162 2864 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: loading cluster ID for follower catalog manager: success
I20260812 19:54:22.741847 2864 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: acquiring CA information for follower catalog manager: success
I20260812 19:54:22.744215 2864 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:22.756829 2770 master_runner.cc:187] Error getting master registration for 127.25.14.61:46307: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46307: connect: Connection refused (error 111),
I20260812 19:54:22.760814 2770 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:54:23.076529 2851 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:23.076967 2851 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:23.077111 2851 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:23.086218 2851 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:23.086365 2851 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:23.086494 2851 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:23.086580 2851 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:23.098274 2851 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:23.100265 2851 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:23.102375 2851 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:23.111546 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:23.111872 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:23.112784 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:23.113981 2851 server_base.cc:1061] running on GCE node
I20260812 19:54:23.115042 2851 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:23.115238 2851 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:23.115348 2851 hybrid_clock.cc:648] HybridClock initialized: now 1786564463115322 us; error 0 us; skew 500 ppm
I20260812 19:54:23.115777 2851 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:23.119666 2851 webserver.cc:533] Webserver started at http://127.25.14.61:39355/ using document root <none> and password file <none>
I20260812 19:54:23.120826 2851 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:23.120990 2851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:23.129315 2851 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 19:54:23.134120 2882 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:23.135864 2851 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 19:54:23.136098 2851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "ee49622bf94e4b82bdd6a59bf585b581"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:23.136924 2851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:23.163314 2851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:23.164372 2851 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:23.164712 2851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:23.187078 2851 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46307
I20260812 19:54:23.187086 2944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46307 every 8 connection(s)
I20260812 19:54:23.188927 2851 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:54:23.192193 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2851
I20260812 19:54:23.192409 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:54:23.193392 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37367
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
I20260812 19:54:23.197110 2945 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:23.200919 2945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Bootstrap starting.
I20260812 19:54:23.225723 2945 log.cc:826] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:23.238791 2945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:23.239585 2945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Bootstrap complete.
I20260812 19:54:23.250779 2945 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:23.251876 2945 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee49622bf94e4b82bdd6a59bf585b581, State: Initialized, Role: FOLLOWER
I20260812 19:54:23.252899 2945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:23.254163 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:23.254492 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:23.255594 2945 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:23.267772 2960 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:23.268004 2960 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: loading cluster ID for follower catalog manager: success
I20260812 19:54:23.272301 2960 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: acquiring CA information for follower catalog manager: success
I20260812 19:54:23.275614 2960 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:23.306013 2851 master_runner.cc:187] Error getting master registration for 127.25.14.60:37367: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37367: connect: Connection refused (error 111),
I20260812 19:54:23.312096 2851 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:54:23.607362 2947 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:23.607726 2947 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:23.607928 2947 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:23.616879 2947 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:23.617009 2947 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:23.617117 2947 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:23.617197 2947 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:23.628438 2947 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:23.630445 2947 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:23.632422 2947 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:23.641079 2970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:23.642210 2971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:23.643855 2974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:23.644845 2947 server_base.cc:1061] running on GCE node
I20260812 19:54:23.645820 2947 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:23.646026 2947 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:23.646133 2947 hybrid_clock.cc:648] HybridClock initialized: now 1786564463646108 us; error 0 us; skew 500 ppm
I20260812 19:54:23.646515 2947 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:23.649685 2947 webserver.cc:533] Webserver started at http://127.25.14.60:39117/ using document root <none> and password file <none>
I20260812 19:54:23.650513 2947 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:23.650674 2947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:23.656538 2947 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:54:23.660028 2979 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:23.661768 2947 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:23.661931 2947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
format_stamp: "Formatted at 2026-08-12 19:54:05 on dist-test-slave-p3t5"
I20260812 19:54:23.662668 2947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:23.694495 2947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:23.695465 2947 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:23.695734 2947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:23.717697 2947 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:37367
I20260812 19:54:23.717747 3039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:37367 every 8 connection(s)
I20260812 19:54:23.719537 2947 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:54:23.726353 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2947
I20260812 19:54:23.726570 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:54:23.727998 3040 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:23.732749 3040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Bootstrap starting.
I20260812 19:54:23.752735 3040 log.cc:826] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:23.762867 3040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:23.763422 3040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: Bootstrap complete.
I20260812 19:54:23.770233 3040 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:23.770982 3040 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6073ff6adbcd4bc9950b1c1c38baa0a2, State: Initialized, Role: FOLLOWER
I20260812 19:54:23.771621 3040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:23.772588 3045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:23.772984 3045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:23.773984 3040 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:23.788216 3057 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:23.788379 3057 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: loading cluster ID for follower catalog manager: success
I20260812 19:54:23.792440 3057 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: acquiring CA information for follower catalog manager: success
I20260812 19:54:23.795397 3057 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:23.816030 2947 master_runner.cc:264] No leader master found from master 6073ff6adbcd4bc9950b1c1c38baa0a2
I20260812 19:54:23.816259 2947 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:24.210410 3061 raft_consensus.cc:493] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:54:24.210759 3061 raft_consensus.cc:515] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:24.213033 3061 leader_election.cc:290] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee49622bf94e4b82bdd6a59bf585b581 (127.25.14.61:46307), 6073ff6adbcd4bc9950b1c1c38baa0a2 (127.25.14.60:37367)
I20260812 19:54:24.221019 2915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "841289fe039443ffb918fd0de05b5d05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee49622bf94e4b82bdd6a59bf585b581" is_pre_election: true
I20260812 19:54:24.221088 3010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "841289fe039443ffb918fd0de05b5d05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" is_pre_election: true
I20260812 19:54:24.221737 3010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841289fe039443ffb918fd0de05b5d05 in term 1.
I20260812 19:54:24.221737 2915 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841289fe039443ffb918fd0de05b5d05 in term 1.
I20260812 19:54:24.222517 2789 leader_election.cc:304] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6073ff6adbcd4bc9950b1c1c38baa0a2, 841289fe039443ffb918fd0de05b5d05, ee49622bf94e4b82bdd6a59bf585b581; no voters:
I20260812 19:54:24.222985 3061 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:54:24.223165 3061 raft_consensus.cc:493] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:54:24.223299 3061 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:54:24.227391 3061 raft_consensus.cc:515] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:24.228333 3061 leader_election.cc:290] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [CANDIDATE]: Term 2 election: Requested vote from peers ee49622bf94e4b82bdd6a59bf585b581 (127.25.14.61:46307), 6073ff6adbcd4bc9950b1c1c38baa0a2 (127.25.14.60:37367)
I20260812 19:54:24.228859 2915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "841289fe039443ffb918fd0de05b5d05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee49622bf94e4b82bdd6a59bf585b581"
I20260812 19:54:24.228945 3010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "841289fe039443ffb918fd0de05b5d05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
I20260812 19:54:24.229086 2915 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:54:24.229223 3010 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:54:24.233063 2915 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841289fe039443ffb918fd0de05b5d05 in term 2.
I20260812 19:54:24.233067 3010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841289fe039443ffb918fd0de05b5d05 in term 2.
I20260812 19:54:24.233716 2789 leader_election.cc:304] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6073ff6adbcd4bc9950b1c1c38baa0a2, 841289fe039443ffb918fd0de05b5d05; no voters:
I20260812 19:54:24.234079 3061 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:54:24.234534 3061 raft_consensus.cc:697] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 LEADER]: Becoming Leader. State: Replica: 841289fe039443ffb918fd0de05b5d05, State: Running, Role: LEADER
I20260812 19:54:24.235055 3061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:24.239048 3064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841289fe039443ffb918fd0de05b5d05. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.239408 3064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:24.240192 3066 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:54:24.242401 3066 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:54:24.243233 3066 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:24.243309 3066 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:54:24.244427 3066 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:54:24.245226 3066 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Loaded TSK: 0
I20260812 19:54:24.246331 3066 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:54:24.271466 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
W20260812 19:54:24.650981 3067 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:24.651381 3067 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:24.651518 3067 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:24.660399 3067 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:24.660648 3067 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:24.663604 3010 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 2 FOLLOWER]: Refusing update from remote peer 841289fe039443ffb918fd0de05b5d05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:54:24.664903 3064 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:54:24.670784 3071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841289fe039443ffb918fd0de05b5d05. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.671068 3071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:24.675356 3067 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:24.678500 3067 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:24.681895 3067 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:54:24.683360 3071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.683593 3071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:24.686093 3075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.686462 3075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:24.687110 2915 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Refusing update from remote peer 841289fe039443ffb918fd0de05b5d05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:54:24.688194 3075 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:54:24.696657 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.697542 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:24.697669 3077 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841289fe039443ffb918fd0de05b5d05. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:24.698027 3077 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:54:24.698355 3082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:24.699405 3083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:24.700928 3067 server_base.cc:1061] running on GCE node
W20260812 19:54:24.701318 3087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:24.702193 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:24.702404 3067 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:24.702517 3067 hybrid_clock.cc:648] HybridClock initialized: now 1786564464702490 us; error 0 us; skew 500 ppm
I20260812 19:54:24.702893 3067 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:24.705516 3067 webserver.cc:533] Webserver started at http://127.25.14.1:38653/ using document root <none> and password file <none>
I20260812 19:54:24.706524 3067 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:24.706674 3067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:24.712921 3067 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:54:24.716619 3096 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:24.718631 3067 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:24.718850 3067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:24.720121 3067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:24.758944 3067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:24.759933 3067 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:24.760303 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:24.761451 3067 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:24.763433 3067 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:24.763521 3067 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:24.763656 3067 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:24.763725 3067 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:24.801908 3067 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35927
I20260812 19:54:24.801949 3222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35927 every 8 connection(s)
I20260812 19:54:24.803577 3067 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:24.812181 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3067
I20260812 19:54:24.812469 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:54:24.822113 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
I20260812 19:54:24.846007 3225 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:24.846025 3224 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:24.846531 3224 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:24.847146 3223 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:24.846526 3225 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:24.847292 3223 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:24.848012 3225 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:24.848055 3223 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:24.848089 3224 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:24.850754 2901 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:24.851044 2807 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:24.852123 2998 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:24.853597 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:53517
I20260812 19:54:25.176008 3061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "841289fe039443ffb918fd0de05b5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:25.176350 3061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: LEADER
W20260812 19:54:25.204890 3233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:25.205271 3233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:25.205417 3233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:25.214440 3233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:25.214679 3233 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:54:25.225320 3233 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:25.227396 3233 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:25.229770 3233 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:25.240818 3241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:25.240878 3242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:25.241868 3245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:25.242376 3233 server_base.cc:1061] running on GCE node
I20260812 19:54:25.243155 3233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:25.243364 3233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:25.243477 3233 hybrid_clock.cc:648] HybridClock initialized: now 1786564465243453 us; error 0 us; skew 500 ppm
I20260812 19:54:25.243831 3233 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:25.246289 3233 webserver.cc:533] Webserver started at http://127.25.14.2:37339/ using document root <none> and password file <none>
I20260812 19:54:25.247215 3233 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:25.247377 3233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:25.253372 3233 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:54:25.256597 3250 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.258219 3233 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:25.258414 3233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "d64ce738a6874f12b9543a4063d3bb95"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:25.259227 3233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:25.295487 3233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:25.296622 3233 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:25.296983 3233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:25.298170 3233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:25.300101 3233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:25.300222 3233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.300387 3233 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:25.300472 3233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.338241 3233 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:34805
I20260812 19:54:25.338299 3376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:34805 every 8 connection(s)
I20260812 19:54:25.340013 3233 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:54:25.349305 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3233
I20260812 19:54:25.349562 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:54:25.355232 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--time_source=system_unsync with env {}
I20260812 19:54:25.360800 3379 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:25.361204 3379 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.361931 3378 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:25.362174 3378 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.362929 3378 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:25.364065 3379 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:25.365409 2807 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:34805)
I20260812 19:54:25.365597 2901 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:34805)
I20260812 19:54:25.366664 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:43431
I20260812 19:54:25.369024 3377 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:25.369159 3377 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.369830 3377 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:25.371106 2998 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:34805)
W20260812 19:54:25.744201 3388 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:25.744549 3388 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:25.744624 3388 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:25.753731 3388 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:25.753922 3388 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:54:25.764783 3388 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:25.766642 3388 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:25.768978 3388 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:25.781389 3395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:25.781474 3396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:25.782021 3399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:25.782682 3388 server_base.cc:1061] running on GCE node
I20260812 19:54:25.783365 3388 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:25.783525 3388 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:25.783599 3388 hybrid_clock.cc:648] HybridClock initialized: now 1786564465783578 us; error 0 us; skew 500 ppm
I20260812 19:54:25.783916 3388 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:25.786550 3388 webserver.cc:533] Webserver started at http://127.25.14.3:34953/ using document root <none> and password file <none>
I20260812 19:54:25.787418 3388 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:25.787544 3388 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:25.793929 3388 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:25.797050 3404 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.798796 3388 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:25.798978 3388 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "c2d51bd329c04985956c7593847cbc63"
format_stamp: "Formatted at 2026-08-12 19:54:07 on dist-test-slave-p3t5"
I20260812 19:54:25.799763 3388 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:25.815071 3388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:25.816061 3388 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:25.816337 3388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:25.817386 3388 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:25.819386 3388 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:25.819474 3388 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.819567 3388 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:25.819613 3388 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:25.856523 3224 heartbeater.cc:499] Master 127.25.14.62:40607 was elected leader, sending a full tablet report...
I20260812 19:54:25.859027 3388 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:37617
I20260812 19:54:25.859045 3532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:37617 every 8 connection(s)
I20260812 19:54:25.860813 3388 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:54:25.870010 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3388
I20260812 19:54:25.870306 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:54:25.891418 3535 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:25.891534 3533 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:25.891881 3533 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.891884 3535 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.891964 3536 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:25.892186 3536 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:25.893260 3535 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:25.893299 3536 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:25.893338 3533 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:25.895285 2807 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:37617)
I20260812 19:54:25.895394 2901 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:37617)
I20260812 19:54:25.895699 2998 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:37617)
I20260812 19:54:25.896514 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:53355
I20260812 19:54:25.900621 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:25.908773 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:25.909024 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2770
I20260812 19:54:25.910043 2843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:26.022569 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2770
I20260812 19:54:26.053658 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--webserver_interface=127.25.14.62
--webserver_port=32909
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367 with env {}
I20260812 19:54:26.370011 3378 heartbeater.cc:499] Master 127.25.14.62:40607 was elected leader, sending a full tablet report...
W20260812 19:54:26.371987 3378 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:40607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:40607: connect: Connection refused (error 111)
W20260812 19:54:26.441395 3546 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:26.441787 3546 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:26.441872 3546 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:26.450843 3546 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:26.450948 3546 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:26.451006 3546 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:26.451054 3546 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:26.462080 3546 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=32909
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:26.463935 3546 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:26.466178 3546 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:26.475256 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:26.475270 3554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:26.476012 3556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:26.476308 3546 server_base.cc:1061] running on GCE node
I20260812 19:54:26.477159 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:26.477329 3546 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:26.477406 3546 hybrid_clock.cc:648] HybridClock initialized: now 1786564466477386 us; error 0 us; skew 500 ppm
I20260812 19:54:26.477787 3546 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:26.480832 3546 webserver.cc:533] Webserver started at http://127.25.14.62:32909/ using document root <none> and password file <none>
I20260812 19:54:26.481720 3546 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:26.481835 3546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:26.487958 3546 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:54:26.491236 3562 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:26.492856 3546 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:54:26.493068 3546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "841289fe039443ffb918fd0de05b5d05"
format_stamp: "Formatted at 2026-08-12 19:54:04 on dist-test-slave-p3t5"
I20260812 19:54:26.493961 3546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:26.529333 3546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:26.530493 3546 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:26.530833 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:26.553541 3546 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40607
I20260812 19:54:26.553576 3622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40607 every 8 connection(s)
I20260812 19:54:26.555610 3546 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:26.563129 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3546
I20260812 19:54:26.563535 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3067
I20260812 19:54:26.566876 3623 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:26.570729 3623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap starting.
I20260812 19:54:26.574296 3218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:26.595731 3623 log.cc:826] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:26.608479 3623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:26.609201 3623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: Bootstrap complete.
I20260812 19:54:26.619928 3623 raft_consensus.cc:359] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:26.620985 3623 raft_consensus.cc:740] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841289fe039443ffb918fd0de05b5d05, State: Initialized, Role: FOLLOWER
I20260812 19:54:26.622000 3623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:26.623031 3631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:26.623382 3631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:26.624387 3623 sys_catalog.cc:565] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:26.635831 3643 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:26.635973 3643 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: loading cluster ID for follower catalog manager: success
I20260812 19:54:26.639180 3643 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: acquiring CA information for follower catalog manager: success
I20260812 19:54:26.642307 3643 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:26.691567 3546 master_runner.cc:264] No leader master found from master 841289fe039443ffb918fd0de05b5d05
I20260812 19:54:26.691769 3546 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:26.694281 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3067
I20260812 19:54:26.719648 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35927
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=38653
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
I20260812 19:54:26.899271 3535 heartbeater.cc:499] Master 127.25.14.62:40607 was elected leader, sending a full tablet report...
I20260812 19:54:26.911084 3583 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c2d51bd329c04985956c7593847cbc63" instance_seqno: 1786564465849143) as {username='slave'} at 127.25.14.3:51041; Asking this server to re-register.
I20260812 19:54:26.912273 3535 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:26.914342 3583 ts_manager.cc:194] Registered new tserver with Master: c2d51bd329c04985956c7593847cbc63 (127.25.14.3:37617)
I20260812 19:54:27.078298 3654 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 841289fe039443ffb918fd0de05b5d05)
I20260812 19:54:27.078559 3654 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:27.080711 3654 leader_election.cc:290] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607), 6073ff6adbcd4bc9950b1c1c38baa0a2 (127.25.14.60:37367)
I20260812 19:54:27.089175 3594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee49622bf94e4b82bdd6a59bf585b581" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "841289fe039443ffb918fd0de05b5d05" is_pre_election: true
I20260812 19:54:27.089890 3594 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee49622bf94e4b82bdd6a59bf585b581 in term 2.
I20260812 19:54:27.090256 3010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee49622bf94e4b82bdd6a59bf585b581" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" is_pre_election: true
I20260812 19:54:27.090778 2884 leader_election.cc:304] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841289fe039443ffb918fd0de05b5d05, ee49622bf94e4b82bdd6a59bf585b581; no voters:
I20260812 19:54:27.091223 3654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:54:27.091372 3654 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 841289fe039443ffb918fd0de05b5d05)
I20260812 19:54:27.091536 3654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:54:27.094446 3654 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:27.095113 3654 leader_election.cc:290] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [CANDIDATE]: Term 3 election: Requested vote from peers 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607), 6073ff6adbcd4bc9950b1c1c38baa0a2 (127.25.14.60:37367)
I20260812 19:54:27.095767 3010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee49622bf94e4b82bdd6a59bf585b581" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2"
I20260812 19:54:27.095865 3594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee49622bf94e4b82bdd6a59bf585b581" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "841289fe039443ffb918fd0de05b5d05"
I20260812 19:54:27.096088 3594 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:54:27.100600 3594 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee49622bf94e4b82bdd6a59bf585b581 in term 3.
I20260812 19:54:27.101133 2884 leader_election.cc:304] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 841289fe039443ffb918fd0de05b5d05, ee49622bf94e4b82bdd6a59bf585b581; no voters: 6073ff6adbcd4bc9950b1c1c38baa0a2
I20260812 19:54:27.101552 3654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:54:27.102140 3654 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [term 3 LEADER]: Becoming Leader. State: Replica: ee49622bf94e4b82bdd6a59bf585b581, State: Running, Role: LEADER
I20260812 19:54:27.102626 3654 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:27.104647 3657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee49622bf94e4b82bdd6a59bf585b581. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.105000 3657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:27.105849 3659 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:54:27.108274 3659 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:54:27.109163 3659 catalog_manager.cc:1295] Loaded cluster ID: 22f1e22797b54ce0b855071277f45f5e
I20260812 19:54:27.109268 3659 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:54:27.110641 3659 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:54:27.111498 3659 catalog_manager.cc:6103] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581: Loaded TSK: 0
I20260812 19:54:27.112440 3659 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260812 19:54:27.118897 3651 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:27.119410 3651 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:27.119565 3651 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:27.128659 3651 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:27.128835 3651 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:27.139247 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35927
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38653
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:27.141072 3651 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:27.143260 3651 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:27.154582 3665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:27.155208 3664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:27.155905 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:27.156666 3651 server_base.cc:1061] running on GCE node
I20260812 19:54:27.157341 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:27.157531 3651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:27.157665 3651 hybrid_clock.cc:648] HybridClock initialized: now 1786564467157605 us; error 0 us; skew 500 ppm
I20260812 19:54:27.158016 3651 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:27.160593 3651 webserver.cc:533] Webserver started at http://127.25.14.1:38653/ using document root <none> and password file <none>
I20260812 19:54:27.161443 3651 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:27.161592 3651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:27.167603 3651 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:54:27.170650 3672 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:27.172048 3651 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:54:27.172242 3651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:27.173023 3651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:27.192698 3651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:27.193753 3651 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:27.194070 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:27.195122 3651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:27.197235 3651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:27.197346 3651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:27.197501 3651 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:27.197578 3651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:27.234850 3651 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35927
I20260812 19:54:27.234943 3805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35927 every 8 connection(s)
I20260812 19:54:27.236479 3651 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:27.242434 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3651
I20260812 19:54:27.255817 3806 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:27.256255 3806 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:27.256814 3807 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:27.256975 3807 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:27.257565 3808 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46307
I20260812 19:54:27.257700 3807 heartbeater.cc:507] Master 127.25.14.62:40607 requested a full tablet report, sending...
I20260812 19:54:27.257920 3806 heartbeater.cc:507] Master 127.25.14.60:37367 requested a full tablet report, sending...
I20260812 19:54:27.257746 3808 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:27.258705 3808 heartbeater.cc:507] Master 127.25.14.61:46307 requested a full tablet report, sending...
I20260812 19:54:27.259959 3583 ts_manager.cc:194] Registered new tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:27.260079 2901 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:27.260185 2998 ts_manager.cc:194] Re-registered known tserver with Master: c13c0f444abd4d8d972a2961b3440c5e (127.25.14.1:35927)
I20260812 19:54:27.262179 2901 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:55991
I20260812 19:54:27.323642 3815 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 841289fe039443ffb918fd0de05b5d05)
I20260812 19:54:27.323902 3815 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } }
I20260812 19:54:27.326308 3815 leader_election.cc:290] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 841289fe039443ffb918fd0de05b5d05 (127.25.14.62:40607), ee49622bf94e4b82bdd6a59bf585b581 (127.25.14.61:46307)
I20260812 19:54:27.327509 2915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee49622bf94e4b82bdd6a59bf585b581" is_pre_election: true
I20260812 19:54:27.339059 3594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "841289fe039443ffb918fd0de05b5d05" is_pre_election: true
I20260812 19:54:27.339530 3594 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6073ff6adbcd4bc9950b1c1c38baa0a2 in current term 3: Already voted for candidate ee49622bf94e4b82bdd6a59bf585b581 in this term.
I20260812 19:54:27.340534 2981 leader_election.cc:304] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6073ff6adbcd4bc9950b1c1c38baa0a2; no voters: 841289fe039443ffb918fd0de05b5d05, ee49622bf94e4b82bdd6a59bf585b581
I20260812 19:54:27.341070 3815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:54:27.343504 3815 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:54:27.380869 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3651
I20260812 19:54:27.382513 3378 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40607
I20260812 19:54:27.382799 3378 heartbeater.cc:499] Master 127.25.14.62:40607 was elected leader, sending a full tablet report...
I20260812 19:54:27.384068 3583 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d64ce738a6874f12b9543a4063d3bb95" instance_seqno: 1786564465329351) as {username='slave'} at 127.25.14.2:43375; Asking this server to re-register.
I20260812 19:54:27.385061 3378 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:27.386276 3583 ts_manager.cc:194] Registered new tserver with Master: d64ce738a6874f12b9543a4063d3bb95 (127.25.14.2:34805)
I20260812 19:54:27.393576 3801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:27.482806 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3651
I20260812 19:54:27.508481 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35927
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=38653
--webserver_interface=127.25.14.1
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260812 19:54:27.618928 3594 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [term 3 FOLLOWER]: Refusing update from remote peer ee49622bf94e4b82bdd6a59bf585b581: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:54:27.619941 3826 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:54:27.624213 3829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee49622bf94e4b82bdd6a59bf585b581. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.624552 3829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:27.626245 3826 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.626510 3826 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:27.627583 3010 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [term 3 FOLLOWER]: Refusing update from remote peer ee49622bf94e4b82bdd6a59bf585b581: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:54:27.632992 3834 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:54:27.633309 3829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.633528 3829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841289fe039443ffb918fd0de05b5d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:27.639447 3815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee49622bf94e4b82bdd6a59bf585b581. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.639808 3815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:27.639355 3840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:27.640118 3840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6073ff6adbcd4bc9950b1c1c38baa0a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:54:27.886768 3825 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:27.887112 3825 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:27.887300 3825 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:54:27.887390 3825 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:27.896090 3825 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:27.896354 3825 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:27.907862 3825 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35927
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38653
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:27.909768 3825 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:27.911834 3825 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:27.923341 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:27.923835 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:27.924060 3850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:27.925475 3825 server_base.cc:1061] running on GCE node
I20260812 19:54:27.926308 3825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:27.926508 3825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:27.926636 3825 hybrid_clock.cc:648] HybridClock initialized: now 1786564467926594 us; error 0 us; skew 500 ppm
I20260812 19:54:27.926997 3825 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:27.929556 3825 webserver.cc:533] Webserver started at http://127.25.14.1:38653/ using document root <none> and password file <none>
I20260812 19:54:27.930552 3825 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:27.930721 3825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:27.936766 3825 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 19:54:27.939874 3857 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:27.941372 3825 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 19:54:27.941576 3825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:27.942464 3825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:27.969759 3825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:27.970847 3825 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:27.971176 3825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:27.972254 3825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:27.974086 3825 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:27.974200 3825 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:27.974339 3825 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:27.974431 3825 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:54:27.974812 3825 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:54:27.974949 3825 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:54:27.975049 3825 fault_injection.cc:59] injecting fault for kudu (pid 3825): FLAGS_fault_before_start (process will exit)
I20260812 19:54:28.002359 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35927
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--webserver_port=38653
--webserver_interface=127.25.14.1
--time_source=system_unsync with env {}
I20260812 19:54:28.067502 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ee49622bf94e4b82bdd6a59bf585b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "841289fe039443ffb918fd0de05b5d05" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40607 } } peers { permanent_uuid: "ee49622bf94e4b82bdd6a59bf585b581" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46307 } } peers { permanent_uuid: "6073ff6adbcd4bc9950b1c1c38baa0a2" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37367 } } }
I20260812 19:54:28.067873 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49622bf94e4b82bdd6a59bf585b581 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:28.372339 3379 heartbeater.cc:499] Master 127.25.14.61:46307 was elected leader, sending a full tablet report...
W20260812 19:54:28.385932 3865 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:28.386299 3865 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:28.386394 3865 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:28.395645 3865 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:28.395838 3865 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:28.406418 3865 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35927
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38653
--tserver_master_addrs=127.25.14.62:40607,127.25.14.61:46307,127.25.14.60:37367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:28.408456 3865 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:28.410697 3865 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:28.422205 3874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:28.422206 3872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:28.422917 3871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:28.423173 3865 server_base.cc:1061] running on GCE node
I20260812 19:54:28.424081 3865 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:54:28.424325 3865 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:54:28.424477 3865 hybrid_clock.cc:648] HybridClock initialized: now 1786564468424448 us; error 0 us; skew 500 ppm
I20260812 19:54:28.424870 3865 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:28.427791 3865 webserver.cc:533] Webserver started at http://127.25.14.1:38653/ using document root <none> and password file <none>
I20260812 19:54:28.428815 3865 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:28.428975 3865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:28.435397 3865 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 19:54:28.438611 3881 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:28.440182 3865 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:54:28.440423 3865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "c13c0f444abd4d8d972a2961b3440c5e"
format_stamp: "Formatted at 2026-08-12 19:54:06 on dist-test-slave-p3t5"
I20260812 19:54:28.441249 3865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:28.466256 3865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:28.467401 3865 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:28.467753 3865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:28.469045 3865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:28.471174 3865 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:28.471258 3865 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:28.471351 3865 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:28.471398 3865 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:28.509334 3865 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35927
I20260812 19:54:28.509429 4010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35927 every 8 connection(s)
I20260812 19:54:28.511106 3865 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:28.521602 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3865
I20260812 19:54:28.522050 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3865
I20260812 19:54:28.553608 4011 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37367
I20260812 19:54:28.554098 4011 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:28.554277 4004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:28.671936 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3865
I20260812 19:54:28.697570 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3233
W20260812 19:54:28.698563 4029 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:54:28.535026 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:47487 (local address 127.25.14.62:40607)
0812 19:54:28.547568 (+ 12542us) server_negotiation.cc:207] Beginning negotiation
0812 19:54:28.547578 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:54:28.548405 (+ 827us) server_negotiation.cc:408] Connection header received
0812 19:54:28.548612 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:54:28.548618 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:54:28.548694 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:54:28.548826 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:54:28.567717 (+ 18891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:54:28.569958 (+ 2241us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:54:28.675749 (+105791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:54:28.676143 (+ 394us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:54:28.697418 (+ 21275us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:47487: BlockingRecv error: recv got EOF from 127.25.14.1:47487 (error 108)
Metrics: {"server-negotiator.queue_time_us":12381,"thread_start_us":12567,"threads_started":1}
I20260812 19:54:28.706369 3372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:28.800737 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3233
I20260812 19:54:28.826366 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3388
I20260812 19:54:28.835551 3528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:28.921973 3536 heartbeater.cc:499] Master 127.25.14.61:46307 was elected leader, sending a full tablet report...
I20260812 19:54:28.926199 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3388
I20260812 19:54:28.949025 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3546
I20260812 19:54:28.956144 3618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:29.065872 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3546
I20260812 19:54:29.092720 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2851
I20260812 19:54:29.094129 2937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:29.208182 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2851
I20260812 19:54:29.238065 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2947
I20260812 19:54:29.239423 3034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:29.347164 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2947
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/3 (42301 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/4
2026-08-12T19:54:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:54:29Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:54:37: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:54:46.285185 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
W20260812 19:54:46.656973 4365 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:46.657357 4365 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:46.657436 4365 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:46.666186 4365 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:46.666303 4365 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:46.666363 4365 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:46.666411 4365 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:46.676970 4365 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:46.678773 4365 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:46.680716 4365 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:46.689930 4371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:46.689960 4370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:46.690390 4374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:46.691200 4365 server_base.cc:1061] running on GCE node
I20260812 19:54:46.692080 4365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:46.693678 4365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:46.694934 4365 hybrid_clock.cc:648] HybridClock initialized: now 1786564486694863 us; error 66 us; skew 500 ppm
I20260812 19:54:46.695338 4365 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:46.697853 4365 webserver.cc:533] Webserver started at http://127.25.14.62:45109/ using document root <none> and password file <none>
I20260812 19:54:46.698706 4365 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:46.698855 4365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:46.699266 4365 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:46.702016 4365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:54:46.703017 4365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:54:46.709420 4365 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:54:46.713553 4380 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:46.715201 4365 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:46.715371 4365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:54:46.715601 4365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:46.770413 4365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:46.771384 4365 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:46.771651 4365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:46.794476 4365 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38107
I20260812 19:54:46.794495 4438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38107 every 8 connection(s)
I20260812 19:54:46.796214 4365 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:46.800993 4439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:46.805433 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4365
I20260812 19:54:46.805701 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:54:46.807166 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:39035
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
I20260812 19:54:46.809724 4439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:46.834610 4439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:46.835913 4384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
W20260812 19:54:46.837723 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
I20260812 19:54:46.887089 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } attempt: 1
W20260812 19:54:46.889417 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
I20260812 19:54:46.957762 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } attempt: 2
W20260812 19:54:46.960187 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
I20260812 19:54:47.051493 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } attempt: 3
W20260812 19:54:47.054131 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
W20260812 19:54:47.185326 4443 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:47.185822 4443 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:47.185958 4443 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:47.194682 4443 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:47.194855 4443 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:47.194974 4443 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:47.195065 4443 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:47.197412 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } attempt: 4
W20260812 19:54:47.199868 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111)
I20260812 19:54:47.206881 4443 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:47.208832 4443 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:47.211040 4443 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:47.220338 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:47.220734 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:47.222956 4455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:47.223500 4443 server_base.cc:1061] running on GCE node
I20260812 19:54:47.224421 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:47.226058 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:47.227272 4443 hybrid_clock.cc:648] HybridClock initialized: now 1786564487227201 us; error 62 us; skew 500 ppm
I20260812 19:54:47.227625 4443 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:47.230111 4443 webserver.cc:533] Webserver started at http://127.25.14.61:42391/ using document root <none> and password file <none>
I20260812 19:54:47.230939 4443 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:47.231045 4443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:47.231384 4443 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:47.236222 4443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "c55730a891034fc1a14be12c5c4fcad5"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.237116 4443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "c55730a891034fc1a14be12c5c4fcad5"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.243425 4443 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 19:54:47.247591 4461 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:47.249452 4443 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:54:47.249717 4443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "c55730a891034fc1a14be12c5c4fcad5"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.249969 4443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:47.269965 4443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:47.270915 4443 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:47.271169 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:47.294265 4443 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:39035
I20260812 19:54:47.294301 4517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:39035 every 8 connection(s)
I20260812 19:54:47.296119 4443 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:54:47.298569 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4443
I20260812 19:54:47.298764 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:54:47.300230 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42935
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
I20260812 19:54:47.302037 4518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:47.311231 4518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:47.326409 4518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:47.336884 4518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:47.338119 4464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
W20260812 19:54:47.339994 4518 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.389458 4518 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 1
W20260812 19:54:47.391855 4518 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.460168 4518 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 2
W20260812 19:54:47.462531 4518 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.499146 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } attempt: 5
I20260812 19:54:47.506826 4439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:54:47.508791 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.553879 4518 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 3
W20260812 19:54:47.556229 4518 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.560026 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 1
W20260812 19:54:47.562182 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.630458 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 2
W20260812 19:54:47.632828 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
W20260812 19:54:47.676460 4520 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:47.676857 4520 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:47.677013 4520 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:47.685691 4520 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:47.685801 4520 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:47.685947 4520 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:47.686026 4520 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:47.696708 4520 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:47.698645 4520 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:47.699530 4518 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 4
I20260812 19:54:47.700836 4520 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:47.701721 4518 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
W20260812 19:54:47.709965 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:47.709965 4529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:47.710819 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:47.711241 4520 server_base.cc:1061] running on GCE node
I20260812 19:54:47.712021 4520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:47.713527 4520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:47.714763 4520 hybrid_clock.cc:648] HybridClock initialized: now 1786564487714728 us; error 31 us; skew 500 ppm
I20260812 19:54:47.715138 4520 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:47.717327 4520 webserver.cc:533] Webserver started at http://127.25.14.60:44339/ using document root <none> and password file <none>
I20260812 19:54:47.718192 4520 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:47.718336 4520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:47.718802 4520 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:47.721356 4520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.722259 4520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.727813 4520 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:54:47.731621 4538 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:47.733042 4520 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 19:54:47.733232 4520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:54:47.733489 4520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:47.739097 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 3
W20260812 19:54:47.741406 4439 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111)
I20260812 19:54:47.751223 4520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:47.752190 4520 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:47.752499 4520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:47.774953 4520 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:42935
I20260812 19:54:47.774978 4599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:42935 every 8 connection(s)
I20260812 19:54:47.776732 4520 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:54:47.780062 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4520
I20260812 19:54:47.780339 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:54:47.782063 4600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:54:47.791035 4600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:47.806324 4600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:47.812975 4600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:54:47.824476 4600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Bootstrap starting.
I20260812 19:54:47.828636 4600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:47.830080 4600 log.cc:826] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:47.832901 4600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: No bootstrap required, opened a new log
I20260812 19:54:47.838482 4600 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:47.838811 4600 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:47.838967 4600 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0003cb21d7ab4025aa0e3cddf7437aa7, State: Initialized, Role: FOLLOWER
I20260812 19:54:47.839627 4600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:47.840718 4606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:47.841236 4606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:47.842155 4600 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:54:47.852705 4617 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:54:47.852986 4617 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:54:47.850030 4520 master_runner.cc:187] Error getting master registration for 127.25.14.62:38107: OK, instance_id { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" instance_seqno: 1786564486788419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:54:47.855948 4520 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:47.918627 4439 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 4
I20260812 19:54:47.931038 4439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap starting.
I20260812 19:54:47.935474 4439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:47.937055 4439 log.cc:826] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:47.940204 4439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: No bootstrap required, opened a new log
I20260812 19:54:47.945848 4439 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:47.946118 4439 raft_consensus.cc:385] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:47.946292 4439 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e6b1432b644b979baf2cd507c498d2, State: Initialized, Role: FOLLOWER
I20260812 19:54:47.946954 4439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:47.948098 4620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:47.948421 4620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:47.949278 4439 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:54:47.961580 4632 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:54:47.961855 4632 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:54:47.960598 4365 master_runner.cc:187] Error getting master registration for 127.25.14.61:39035: OK, instance_id { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" instance_seqno: 1786564487288077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:54:47.965461 4365 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:54:48.000993 4518 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } attempt: 5
I20260812 19:54:48.004091 4606 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:54:48.004441 4606 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
W20260812 19:54:48.004801 4548 tablet.cc:2406] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:54:48.007046 4606 leader_election.cc:290] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107), c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035)
I20260812 19:54:48.007591 4411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e6b1432b644b979baf2cd507c498d2" is_pre_election: true
I20260812 19:54:48.008224 4411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0003cb21d7ab4025aa0e3cddf7437aa7 in term 0.
I20260812 19:54:48.009055 4543 leader_election.cc:304] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0003cb21d7ab4025aa0e3cddf7437aa7, 14e6b1432b644b979baf2cd507c498d2; no voters:
I20260812 19:54:48.009698 4606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:54:48.009865 4606 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:54:48.009994 4606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:54:48.020502 4606 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:48.021667 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Bootstrap starting.
W20260812 19:54:48.024617 4386 tablet.cc:2406] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:54:48.028425 4411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e6b1432b644b979baf2cd507c498d2"
I20260812 19:54:48.028689 4411 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:54:48.032429 4411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0003cb21d7ab4025aa0e3cddf7437aa7 in term 1.
I20260812 19:54:48.032778 4606 leader_election.cc:290] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 1 election: Requested vote from peers 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107), c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035)
I20260812 19:54:48.033145 4543 leader_election.cc:304] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0003cb21d7ab4025aa0e3cddf7437aa7, 14e6b1432b644b979baf2cd507c498d2; no voters:
I20260812 19:54:48.034734 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Neither blocks nor log segments found. Creating new log.
I20260812 19:54:48.036288 4518 log.cc:826] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:48.037029 4606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:54:48.043187 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: No bootstrap required, opened a new log
I20260812 19:54:48.049315 4606 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 LEADER]: Becoming Leader. State: Replica: 0003cb21d7ab4025aa0e3cddf7437aa7, State: Running, Role: LEADER
I20260812 19:54:48.050010 4606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:48.060338 4518 raft_consensus.cc:359] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:48.060770 4518 raft_consensus.cc:385] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:54:48.060917 4518 raft_consensus.cc:740] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55730a891034fc1a14be12c5c4fcad5, State: Initialized, Role: FOLLOWER
I20260812 19:54:48.064777 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:48.067584 4635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0003cb21d7ab4025aa0e3cddf7437aa7. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.068739 4635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:48.069921 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.070371 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:48.071170 4638 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:54:48.071720 4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55730a891034fc1a14be12c5c4fcad5"
I20260812 19:54:48.072228 4491 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:54:48.073707 4638 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:54:48.071775 4492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55730a891034fc1a14be12c5c4fcad5" is_pre_election: true
I20260812 19:54:48.078012 4491 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0003cb21d7ab4025aa0e3cddf7437aa7 in term 1.
I20260812 19:54:48.071693 4518 sys_catalog.cc:565] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:48.090847 4411 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Refusing update from remote peer 0003cb21d7ab4025aa0e3cddf7437aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:54:48.091022 4491 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Refusing update from remote peer 0003cb21d7ab4025aa0e3cddf7437aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:54:48.091972 4606 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:54:48.092818 4606 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:54:48.107679 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0003cb21d7ab4025aa0e3cddf7437aa7. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.107980 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:48.121959 4639 mvcc.cc:204] Tried to move back new op lower bound from 7317768143206854656 to 7317768143059456000. Current Snapshot: MvccSnapshot[applied={T|T < 7317768143206854656}]
I20260812 19:54:48.131891 4644 mvcc.cc:204] Tried to move back new op lower bound from 7317768143206854656 to 7317768143059456000. Current Snapshot: MvccSnapshot[applied={T|T < 7317768143206854656}]
I20260812 19:54:48.135974 4606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.136372 4606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:48.136883 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.137111 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:48.141662 4620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0003cb21d7ab4025aa0e3cddf7437aa7. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.141969 4620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:48.142606 4638 catalog_manager.cc:1383] Generated new cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:54:48.142724 4638 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:54:48.150480 4620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.150750 4620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:48.154629 4635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:48.155989 4635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:48.160202 4646 mvcc.cc:204] Tried to move back new op lower bound from 7317768143206854656 to 7317768143059456000. Current Snapshot: MvccSnapshot[applied={T|T < 7317768143206854656 or (T in {7317768143206854656})}]
I20260812 19:54:48.166320 4638 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:54:48.168179 4638 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:54:48.197867 4638 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Generated new TSK 0
I20260812 19:54:48.199496 4638 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:54:48.222525 4661 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:54:48.222663 4661 catalog_manager.cc:1588] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: loading cluster ID for follower catalog manager: success
I20260812 19:54:48.230969 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:54:48.234014 4661 catalog_manager.cc:1610] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: acquiring CA information for follower catalog manager: success
I20260812 19:54:48.237071 4661 catalog_manager.cc:1638] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:54:48.719389 4663 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:48.719807 4663 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:48.719950 4663 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:48.728938 4663 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:48.729189 4663 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:48.739981 4663 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:48.741864 4663 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:48.743991 4663 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:48.755904 4671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:48.755787 4670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:48.761888 4663 server_base.cc:1061] running on GCE node
W20260812 19:54:48.762835 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:48.763589 4663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:48.764706 4663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:48.765957 4663 hybrid_clock.cc:648] HybridClock initialized: now 1786564488765895 us; error 55 us; skew 500 ppm
I20260812 19:54:48.766315 4663 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:48.768894 4663 webserver.cc:533] Webserver started at http://127.25.14.1:44435/ using document root <none> and password file <none>
I20260812 19:54:48.769867 4663 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:48.770037 4663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:48.770428 4663 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:48.773204 4663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:48.774127 4663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:48.780946 4663 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:54:48.785559 4683 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:48.787427 4663 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:48.787665 4663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:48.787947 4663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:48.807219 4663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:48.808229 4663 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:48.808557 4663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:48.809751 4663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:48.811648 4663 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:48.811782 4663 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:48.811932 4663 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:48.812031 4663 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:48.849730 4663 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38035
I20260812 19:54:48.849836 4816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38035 every 8 connection(s)
I20260812 19:54:48.851560 4663 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:48.860368 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4663
I20260812 19:54:48.860600 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:54:48.867000 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:54:48.872537 4817 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:54:48.872995 4817 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:48.874162 4817 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:54:48.875871 4819 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:54:48.876084 4819 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:48.876624 4819 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:54:48.876905 4561 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:48.878625 4479 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:48.879290 4561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41415
I20260812 19:54:48.884793 4818 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:48.884992 4818 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:48.885422 4818 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:54:48.887897 4399 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:48.963909 4632 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:54:48.964043 4632 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: loading cluster ID for follower catalog manager: success
I20260812 19:54:48.966928 4632 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: acquiring CA information for follower catalog manager: success
I20260812 19:54:48.968806 4632 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:54:49.247387 4830 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:49.247839 4830 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:49.248003 4830 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:49.256950 4830 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:49.257197 4830 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:54:49.268608 4830 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:49.270996 4830 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:49.272970 4830 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:49.283658 4836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:49.283596 4835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:49.284276 4838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:49.285137 4830 server_base.cc:1061] running on GCE node
I20260812 19:54:49.285902 4830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:49.287045 4830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:49.288281 4830 hybrid_clock.cc:648] HybridClock initialized: now 1786564489288216 us; error 64 us; skew 500 ppm
I20260812 19:54:49.288628 4830 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:49.290967 4830 webserver.cc:533] Webserver started at http://127.25.14.2:39901/ using document root <none> and password file <none>
I20260812 19:54:49.291824 4830 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:49.291983 4830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:49.292404 4830 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:49.295262 4830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "36caa9630e634111bdc132b24dcec0fb"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.296088 4830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "36caa9630e634111bdc132b24dcec0fb"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.302667 4830 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 19:54:49.306617 4844 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.308380 4830 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:54:49.308589 4830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "36caa9630e634111bdc132b24dcec0fb"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.308838 4830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:49.332441 4830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:49.333424 4830 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:49.333796 4830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:49.334843 4830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:49.336751 4830 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:49.336882 4830 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.337030 4830 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:49.337126 4830 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.375164 4830 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:45225
I20260812 19:54:49.375303 4977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:45225 every 8 connection(s)
I20260812 19:54:49.376788 4830 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:54:49.383823 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4830
I20260812 19:54:49.384104 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:54:49.392342 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:54:49.398945 4979 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:49.399410 4979 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.400364 4979 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:54:49.402401 4399 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:45225)
I20260812 19:54:49.405838 4980 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:54:49.405974 4980 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.406401 4980 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:54:49.407737 4479 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:45225)
I20260812 19:54:49.417524 4978 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:54:49.417774 4978 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.418273 4978 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:54:49.419363 4561 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:45225)
I20260812 19:54:49.420187 4561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:36485
W20260812 19:54:49.803561 4988 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:49.803958 4988 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:49.804046 4988 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:49.814476 4988 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:49.814742 4988 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:54:49.827145 4988 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:49.829098 4988 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:49.831342 4988 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:49.843545 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:49.843526 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:49.843544 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:49.844934 4988 server_base.cc:1061] running on GCE node
I20260812 19:54:49.845744 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:49.846820 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:49.848069 4988 hybrid_clock.cc:648] HybridClock initialized: now 1786564489848014 us; error 56 us; skew 500 ppm
I20260812 19:54:49.848423 4988 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:49.851342 4988 webserver.cc:533] Webserver started at http://127.25.14.3:33783/ using document root <none> and password file <none>
I20260812 19:54:49.852452 4988 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:49.852607 4988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:49.853065 4988 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:54:49.856591 4988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "3b85ff9460c543a18c7e6011d2c10fb4"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.857383 4988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "3b85ff9460c543a18c7e6011d2c10fb4"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.863229 4988 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 19:54:49.866961 5005 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.868515 4988 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:54:49.868726 4988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "3b85ff9460c543a18c7e6011d2c10fb4"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:54:49.868978 4988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:49.883348 4817 heartbeater.cc:499] Master 127.25.14.60:42935 was elected leader, sending a full tablet report...
I20260812 19:54:49.889379 4988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:49.890527 4988 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:49.890879 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:49.891903 4988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:49.893770 4988 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:49.893870 4988 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.893958 4988 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:49.894019 4988 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:49.931398 4988 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:39611
I20260812 19:54:49.931484 5135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:39611 every 8 connection(s)
I20260812 19:54:49.933030 4988 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:54:49.934863 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4988
I20260812 19:54:49.935094 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:54:49.956336 5139 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:54:49.956849 5139 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.958073 5139 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:54:49.960633 5136 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:54:49.960737 5138 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:49.960825 5136 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.960860 5138 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:49.961437 5136 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:54:49.961426 4479 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:39611)
I20260812 19:54:49.961701 5138 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:54:49.962569 4561 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:39611)
I20260812 19:54:49.962944 4399 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:39611)
I20260812 19:54:49.963625 4561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:45805
I20260812 19:54:49.966048 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:49.974623 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:54:49.974931 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4365
I20260812 19:54:49.975967 4434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:50.089807 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4365
I20260812 19:54:50.118445 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--webserver_interface=127.25.14.62
--webserver_port=45109
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
W20260812 19:54:50.321700 4543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:38107: connect: Connection refused (error 111)
W20260812 19:54:50.323647 4543 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 -> Peer 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107): Couldn't send request to peer 14e6b1432b644b979baf2cd507c498d2. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:38107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:54:50.406469 4979 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:38107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:38107: connect: Connection refused (error 111)
I20260812 19:54:50.422927 4978 heartbeater.cc:499] Master 127.25.14.60:42935 was elected leader, sending a full tablet report...
W20260812 19:54:50.521577 5151 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:50.522006 5151 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:50.522153 5151 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:50.531157 5151 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:54:50.531275 5151 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:50.531363 5151 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:54:50.531445 5151 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:54:50.542490 5151 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=45109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:50.544567 5151 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:50.546813 5151 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:50.556169 5162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:50.557327 5165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:50.557359 5163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:50.557950 5151 server_base.cc:1061] running on GCE node
I20260812 19:54:50.558851 5151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:50.560536 5151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:50.561810 5151 hybrid_clock.cc:648] HybridClock initialized: now 1786564490561737 us; error 56 us; skew 500 ppm
I20260812 19:54:50.562345 5151 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:50.565302 5151 webserver.cc:533] Webserver started at http://127.25.14.62:45109/ using document root <none> and password file <none>
I20260812 19:54:50.566241 5151 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:50.566406 5151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:50.572786 5151 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:54:50.576366 5171 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:50.578027 5151 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:54:50.578227 5151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:54:50.579035 5151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:50.614627 5151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:50.615724 5151 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:50.616065 5151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:50.639900 5151 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38107
I20260812 19:54:50.639931 5231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38107 every 8 connection(s)
I20260812 19:54:50.641884 5151 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:54:50.650189 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5151
I20260812 19:54:50.650333 5233 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:54:50.650614 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4663
I20260812 19:54:50.655860 5233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap starting.
I20260812 19:54:50.661330 4812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:50.681679 5233 log.cc:826] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:54:50.693836 5233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:54:50.694583 5233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap complete.
I20260812 19:54:50.705518 5233 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:50.706614 5233 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e6b1432b644b979baf2cd507c498d2, State: Initialized, Role: FOLLOWER
I20260812 19:54:50.707607 5233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:54:50.708582 5239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:50.708899 5239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:50.710043 5233 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:54:50.721788 5253 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:54:50.722023 5253 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: loading cluster ID for follower catalog manager: success
I20260812 19:54:50.726619 5253 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: acquiring CA information for follower catalog manager: success
I20260812 19:54:50.729703 5253 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:54:50.787793 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4663
I20260812 19:54:50.820791 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38035
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=44435
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:54:50.883989 5152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:50.884361 5152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:50.901928 5239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0003cb21d7ab4025aa0e3cddf7437aa7. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:50.902253 5239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:54:50.966495 5136 heartbeater.cc:499] Master 127.25.14.60:42935 was elected leader, sending a full tablet report...
I20260812 19:54:50.976202 5195 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3b85ff9460c543a18c7e6011d2c10fb4" instance_seqno: 1786564489922656) as {username='slave'} at 127.25.14.3:59633; Asking this server to re-register.
I20260812 19:54:50.977231 5138 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:50.979259 5195 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:39611)
W20260812 19:54:51.214072 5260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:51.214572 5260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:51.214732 5260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:51.223843 5260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:51.224095 5260 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:51.234927 5260 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44435
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:51.236763 5260 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:51.239120 5260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:51.250253 5273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:51.250336 5274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:51.251039 5276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:51.251549 5260 server_base.cc:1061] running on GCE node
I20260812 19:54:51.252238 5260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:51.253340 5260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:51.254606 5260 hybrid_clock.cc:648] HybridClock initialized: now 1786564491254530 us; error 68 us; skew 500 ppm
I20260812 19:54:51.255059 5260 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:51.257577 5260 webserver.cc:533] Webserver started at http://127.25.14.1:44435/ using document root <none> and password file <none>
I20260812 19:54:51.258461 5260 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:51.258639 5260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:51.264861 5260 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:54:51.268095 5282 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:51.269747 5260 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:51.269959 5260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:51.270803 5260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:51.293445 5260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:51.294497 5260 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:51.294854 5260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:51.296192 5260 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:51.298552 5260 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:51.298636 5260 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:51.298815 5260 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:51.298884 5260 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:51.314617 5155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:54:51.314972 5155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: LEADER
I20260812 19:54:51.337817 5260 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38035
I20260812 19:54:51.337841 5418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38035 every 8 connection(s)
I20260812 19:54:51.339654 5260 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:51.340936 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5260
I20260812 19:54:51.390854 5421 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:54:51.391394 5421 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:51.391445 5420 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:51.391589 5420 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:51.392578 5420 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:54:51.393841 5421 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:54:51.394604 5195 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:51.395313 5419 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:54:51.395459 5419 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:51.395984 5419 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:54:51.397193 4479 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:51.397219 4561 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:51.398183 4561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:48867
I20260812 19:54:51.418362 4979 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:51.419781 5195 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "36caa9630e634111bdc132b24dcec0fb" instance_seqno: 1786564489366048) as {username='slave'} at 127.25.14.2:47513; Asking this server to re-register.
I20260812 19:54:51.420452 4979 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:51.421918 5195 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:45225)
I20260812 19:54:51.422528 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5260
I20260812 19:54:51.437058 5413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:51.526335 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5260
I20260812 19:54:51.553791 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38035
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=44435
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260812 19:54:51.925282 5433 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:51.925661 5433 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:51.925781 5433 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:54:51.925836 5433 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:51.936259 5433 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:51.936471 5433 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:51.947268 5433 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44435
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:51.949043 5433 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:51.951113 5433 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:51.962540 5439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:51.962635 5441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:51.962497 5443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:51.963721 5433 server_base.cc:1061] running on GCE node
I20260812 19:54:51.964385 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:51.965463 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:51.966749 5433 hybrid_clock.cc:648] HybridClock initialized: now 1786564491966680 us; error 54 us; skew 500 ppm
I20260812 19:54:51.967208 5433 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:51.970082 5433 webserver.cc:533] Webserver started at http://127.25.14.1:44435/ using document root <none> and password file <none>
I20260812 19:54:51.971112 5433 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:51.971276 5433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:51.977463 5433 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:54:51.980660 5450 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:51.982542 5433 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:54:51.982731 5433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:51.983544 5433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:51.999341 5433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:52.000381 5433 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:52.000711 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:52.001818 5433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:52.003636 5433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:52.003769 5433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:52.003927 5433 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:52.004021 5433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:54:52.004408 5433 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:54:52.004549 5433 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:54:52.004629 5433 fault_injection.cc:59] injecting fault for kudu (pid 5433): FLAGS_fault_before_start (process will exit)
I20260812 19:54:52.032533 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38035
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=44435
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:54:52.408712 5459 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:54:52.409168 5459 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:54:52.409346 5459 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:54:52.418658 5459 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:54:52.418893 5459 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:54:52.430533 5459 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44435
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:54:52.432310 5459 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:54:52.434345 5459 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:54:52.445730 5464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:52.445655 5466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:54:52.445952 5468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:54:52.447652 5459 server_base.cc:1061] running on GCE node
I20260812 19:54:52.448487 5459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:54:52.449515 5459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:54:52.450835 5459 hybrid_clock.cc:648] HybridClock initialized: now 1786564492450754 us; error 70 us; skew 500 ppm
I20260812 19:54:52.451216 5459 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:54:52.453761 5459 webserver.cc:533] Webserver started at http://127.25.14.1:44435/ using document root <none> and password file <none>
I20260812 19:54:52.454612 5459 fs_manager.cc:362] Metadata directory not provided
I20260812 19:54:52.454787 5459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:54:52.461289 5459 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:54:52.464609 5474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:52.466310 5459 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:54:52.466533 5459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:54:52.467545 5459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:54:52.488837 5459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:54:52.489935 5459 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:54:52.490258 5459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:54:52.491458 5459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:54:52.493351 5459 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:54:52.493472 5459 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:54:52.493670 5459 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:54:52.493757 5459 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 19:54:52.532516 5459 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38035
I20260812 19:54:52.532564 5601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38035 every 8 connection(s)
I20260812 19:54:52.534255 5459 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:54:52.542110 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5459
I20260812 19:54:52.542497 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5459
I20260812 19:54:52.562570 5602 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:54:52.562369 5604 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:54:52.563179 5604 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:52.563210 5603 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:54:52.563419 5603 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:52.563692 5602 heartbeater.cc:461] Registering TS with master...
I20260812 19:54:52.564859 5602 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:54:52.565570 5604 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:54:52.566053 5603 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:54:52.568779 4479 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:52.568804 4561 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:52.570079 4561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:46545
I20260812 19:54:52.571069 5195 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:38035)
I20260812 19:54:52.573257 5596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:52.667296 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5459
I20260812 19:54:52.692685 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4830
I20260812 19:54:52.701273 4973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:52.795579 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4830
I20260812 19:54:52.820271 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4988
I20260812 19:54:52.829175 5130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:52.920223 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4988
I20260812 19:54:52.946578 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5151
I20260812 19:54:52.953455 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:53.061307 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5151
I20260812 19:54:53.090101 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4443
I20260812 19:54:53.091401 4513 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 19:54:53.150907 4543 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 -> Peer 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107): Couldn't send request to peer 14e6b1432b644b979baf2cd507c498d2. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:38107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:54:53.199210 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4443
W20260812 19:54:53.226914 4544 connection.cc:573] client connection to 127.25.14.61:39035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:54:53.227341 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4520
W20260812 19:54:53.228346 4544 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 -> Peer c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035): Couldn't send request to peer c55730a891034fc1a14be12c5c4fcad5. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:54:53.228612 4595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:54:53.340761 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4520
Shutting down hive metastore.
2026-08-12T19:54:53Z chronyd exiting
2026-08-12T19:54:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:54:53Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:54:54: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:55:03.491514 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
W20260812 19:55:03.863698 5838 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:03.864172 5838 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:03.864327 5838 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:03.873096 5838 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:03.873261 5838 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:03.873379 5838 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:03.873467 5838 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:03.884281 5838 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:03.886199 5838 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:03.888218 5838 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:03.897789 5843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:03.897871 5844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:03.898427 5846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:03.898818 5838 server_base.cc:1061] running on GCE node
I20260812 19:55:03.899758 5838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:03.901351 5838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:03.902629 5838 hybrid_clock.cc:648] HybridClock initialized: now 1786564503902555 us; error 82 us; skew 500 ppm
I20260812 19:55:03.903127 5838 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:03.905608 5838 webserver.cc:533] Webserver started at http://127.25.14.62:36573/ using document root <none> and password file <none>
I20260812 19:55:03.906553 5838 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:03.906705 5838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:03.912676 5838 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:55:03.916252 5853 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:03.917996 5838 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:03.918239 5838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:55:03.919077 5838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:03.949102 5838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:03.950208 5838 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:03.950557 5838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:03.973316 5838 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38107
I20260812 19:55:03.973336 5910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38107 every 8 connection(s)
I20260812 19:55:03.975244 5838 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:03.981287 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5838
I20260812 19:55:03.981524 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:55:03.982695 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:39035
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
I20260812 19:55:03.983845 5911 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:03.987665 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap starting.
I20260812 19:55:04.005579 5911 log.cc:826] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:04.015581 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:04.016207 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap complete.
I20260812 19:55:04.023658 5911 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:04.024576 5911 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e6b1432b644b979baf2cd507c498d2, State: Initialized, Role: FOLLOWER
I20260812 19:55:04.025484 5911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:04.026531 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:04.026885 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:04.027765 5911 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:04.041229 5927 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:04.041518 5927 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: loading cluster ID for follower catalog manager: success
I20260812 19:55:04.049261 5927 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: acquiring CA information for follower catalog manager: success
I20260812 19:55:04.052657 5927 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:04.067924 5838 master_runner.cc:187] Error getting master registration for 127.25.14.61:39035: Network error: Client connection negotiation failed: client connection to 127.25.14.61:39035: connect: Connection refused (error 111),
I20260812 19:55:04.073875 5838 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:04.376824 5913 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:04.377280 5913 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:04.377424 5913 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:04.386384 5913 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:04.386533 5913 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:04.386662 5913 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:04.386771 5913 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:04.397316 5913 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:04.399156 5913 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:04.401155 5913 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:04.410534 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:04.410588 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:04.412716 5940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:04.413257 5913 server_base.cc:1061] running on GCE node
I20260812 19:55:04.414454 5913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:04.416543 5913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:04.417805 5913 hybrid_clock.cc:648] HybridClock initialized: now 1786564504417732 us; error 88 us; skew 500 ppm
I20260812 19:55:04.418270 5913 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:04.421229 5913 webserver.cc:533] Webserver started at http://127.25.14.61:39075/ using document root <none> and password file <none>
I20260812 19:55:04.422322 5913 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:04.422500 5913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:04.430656 5913 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:55:04.434680 5947 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:04.436530 5913 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 19:55:04.436739 5913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "c55730a891034fc1a14be12c5c4fcad5"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:55:04.437861 5913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:04.483232 5913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:04.484184 5913 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:04.484473 5913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:04.507316 5913 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:39035
I20260812 19:55:04.507330 6004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:39035 every 8 connection(s)
I20260812 19:55:04.509233 5913 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:55:04.513019 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5913
I20260812 19:55:04.513259 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:55:04.514627 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42935
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
I20260812 19:55:04.519174 6005 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:04.525051 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Bootstrap starting.
I20260812 19:55:04.544433 6005 log.cc:826] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:04.555133 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:04.555770 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Bootstrap complete.
I20260812 19:55:04.566766 6005 raft_consensus.cc:359] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:04.567891 6005 raft_consensus.cc:740] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55730a891034fc1a14be12c5c4fcad5, State: Initialized, Role: FOLLOWER
I20260812 19:55:04.569063 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:04.570097 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:04.570430 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:04.571303 6005 sys_catalog.cc:565] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:04.582882 6022 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:04.583127 6022 catalog_manager.cc:1588] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: loading cluster ID for follower catalog manager: success
I20260812 19:55:04.588475 6022 catalog_manager.cc:1610] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: acquiring CA information for follower catalog manager: success
I20260812 19:55:04.591964 6022 catalog_manager.cc:1638] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:04.616588 5913 master_runner.cc:187] Error getting master registration for 127.25.14.60:42935: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42935: connect: Connection refused (error 111),
I20260812 19:55:04.620486 5913 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:04.935983 6007 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:04.936406 6007 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:04.936547 6007 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:04.945487 6007 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:04.945868 6007 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:04.946048 6007 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:04.946151 6007 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:04.957674 6007 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:04.959645 6007 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:04.962141 6007 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:04.971089 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:04.971318 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:04.973009 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:04.973906 6007 server_base.cc:1061] running on GCE node
I20260812 19:55:04.974949 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:04.977991 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:04.979343 6007 hybrid_clock.cc:648] HybridClock initialized: now 1786564504979259 us; error 82 us; skew 500 ppm
I20260812 19:55:04.979776 6007 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:04.982918 6007 webserver.cc:533] Webserver started at http://127.25.14.60:35407/ using document root <none> and password file <none>
I20260812 19:55:04.983987 6007 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:04.984133 6007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:04.991658 6007 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 19:55:04.995254 6044 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:04.996979 6007 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:04.997193 6007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
format_stamp: "Formatted at 2026-08-12 19:54:47 on dist-test-slave-p3t5"
I20260812 19:55:04.998090 6007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:05.013952 6007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:05.015029 6007 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:05.015456 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:05.038197 6007 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:42935
I20260812 19:55:05.038223 6104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:42935 every 8 connection(s)
I20260812 19:55:05.040087 6007 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:55:05.046432 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6007
I20260812 19:55:05.046648 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:55:05.049026 6105 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:05.054189 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Bootstrap starting.
I20260812 19:55:05.076242 6105 log.cc:826] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:05.086274 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:05.086912 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: Bootstrap complete.
I20260812 19:55:05.093853 6105 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:05.094626 6105 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0003cb21d7ab4025aa0e3cddf7437aa7, State: Initialized, Role: FOLLOWER
I20260812 19:55:05.095335 6105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:05.096172 6110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:05.096488 6110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:05.097272 6105 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:05.113034 6124 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:05.113237 6124 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: loading cluster ID for follower catalog manager: success
I20260812 19:55:05.117535 6124 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: acquiring CA information for follower catalog manager: success
I20260812 19:55:05.120774 6124 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:05.139389 6007 master_runner.cc:264] No leader master found from master 0003cb21d7ab4025aa0e3cddf7437aa7
I20260812 19:55:05.139602 6007 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:05.618294 6127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:55:05.618669 6127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:05.620672 6127 leader_election.cc:290] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035), 0003cb21d7ab4025aa0e3cddf7437aa7 (127.25.14.60:42935)
I20260812 19:55:05.628216 6075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14e6b1432b644b979baf2cd507c498d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" is_pre_election: true
I20260812 19:55:05.628247 5978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14e6b1432b644b979baf2cd507c498d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c55730a891034fc1a14be12c5c4fcad5" is_pre_election: true
I20260812 19:55:05.628845 6075 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e6b1432b644b979baf2cd507c498d2 in term 1.
I20260812 19:55:05.628844 5978 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e6b1432b644b979baf2cd507c498d2 in term 1.
I20260812 19:55:05.629580 5857 leader_election.cc:304] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0003cb21d7ab4025aa0e3cddf7437aa7, 14e6b1432b644b979baf2cd507c498d2, c55730a891034fc1a14be12c5c4fcad5; no voters:
I20260812 19:55:05.630119 6127 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:55:05.630280 6127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:55:05.630420 6127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:05.634430 6127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:05.635253 6127 leader_election.cc:290] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [CANDIDATE]: Term 2 election: Requested vote from peers c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035), 0003cb21d7ab4025aa0e3cddf7437aa7 (127.25.14.60:42935)
I20260812 19:55:05.635843 5978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14e6b1432b644b979baf2cd507c498d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c55730a891034fc1a14be12c5c4fcad5"
I20260812 19:55:05.635936 6075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14e6b1432b644b979baf2cd507c498d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
I20260812 19:55:05.636106 5978 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:05.636168 6075 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:05.641708 5978 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e6b1432b644b979baf2cd507c498d2 in term 2.
I20260812 19:55:05.642149 6075 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e6b1432b644b979baf2cd507c498d2 in term 2.
I20260812 19:55:05.642344 5858 leader_election.cc:304] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e6b1432b644b979baf2cd507c498d2, c55730a891034fc1a14be12c5c4fcad5; no voters:
I20260812 19:55:05.642798 6127 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:55:05.643347 6127 raft_consensus.cc:697] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 LEADER]: Becoming Leader. State: Replica: 14e6b1432b644b979baf2cd507c498d2, State: Running, Role: LEADER
I20260812 19:55:05.644346 6127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:05.647385 6130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14e6b1432b644b979baf2cd507c498d2. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:05.647660 6130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:05.648285 6132 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:05.650601 6132 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:05.651458 6132 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:05.651571 6132 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:05.652735 6132 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:05.653559 6132 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Loaded TSK: 0
I20260812 19:55:05.654538 6132 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:55:05.679417 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:55:06.059274 6133 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:06.059679 6133 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:06.059827 6133 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:06.069296 6133 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:06.069564 6133 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:06.070091 6075 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 2 FOLLOWER]: Refusing update from remote peer 14e6b1432b644b979baf2cd507c498d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:55:06.070183 5978 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Refusing update from remote peer 14e6b1432b644b979baf2cd507c498d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:55:06.071276 6127 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:55:06.071754 6130 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:55:06.075505 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14e6b1432b644b979baf2cd507c498d2. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.075811 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:06.076882 6137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14e6b1432b644b979baf2cd507c498d2. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.077185 6137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:06.087448 6137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.087702 6137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:06.087925 6133 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:06.090617 6133 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:06.090684 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.090989 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:06.092108 6127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.092048 6142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "14e6b1432b644b979baf2cd507c498d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:06.092362 6127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:06.092460 6142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:06.093252 6133 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:06.104146 6148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:06.104113 6149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:06.104938 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:06.105506 6133 server_base.cc:1061] running on GCE node
I20260812 19:55:06.106184 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:06.107250 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:06.108506 6133 hybrid_clock.cc:648] HybridClock initialized: now 1786564506108452 us; error 45 us; skew 500 ppm
I20260812 19:55:06.108879 6133 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:06.111366 6133 webserver.cc:533] Webserver started at http://127.25.14.1:35919/ using document root <none> and password file <none>
I20260812 19:55:06.112237 6133 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:06.112406 6133 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:06.118594 6133 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:55:06.121881 6158 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.123535 6133 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:06.123699 6133 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:55:06.124550 6133 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:06.148765 6133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:06.149735 6133 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:06.150018 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:06.151180 6133 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:06.153015 6133 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:06.153095 6133 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.153188 6133 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:06.153234 6133 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.190001 6133 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35641
I20260812 19:55:06.190115 6285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35641 every 8 connection(s)
I20260812 19:55:06.191709 6133 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:06.198870 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6133
I20260812 19:55:06.199110 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:55:06.204754 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:06.218232 6287 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:06.218304 6286 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:55:06.218271 6288 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:55:06.218842 6288 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.218840 6286 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.220113 6288 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:55:06.220829 6286 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:55:06.218838 6287 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.221879 6287 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:55:06.223294 5966 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:06.223305 6065 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:06.223376 5873 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:06.225528 5873 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:42625
W20260812 19:55:06.596603 6296 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:06.596976 6296 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:06.597137 6296 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:06.606257 6296 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:06.606490 6296 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:55:06.618189 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:06.620051 6296 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:06.622184 6296 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:06.632886 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:06.632884 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:06.633302 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:06.634316 6296 server_base.cc:1061] running on GCE node
I20260812 19:55:06.634951 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:06.635867 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:06.637075 6296 hybrid_clock.cc:648] HybridClock initialized: now 1786564506637027 us; error 32 us; skew 500 ppm
I20260812 19:55:06.637425 6296 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:06.639750 6296 webserver.cc:533] Webserver started at http://127.25.14.2:38069/ using document root <none> and password file <none>
I20260812 19:55:06.640596 6296 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:06.640761 6296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:06.646759 6296 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 19:55:06.649896 6312 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.651399 6296 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:55:06.651607 6296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "36caa9630e634111bdc132b24dcec0fb"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:55:06.652490 6296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:06.666749 6296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:06.667732 6296 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:06.668074 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:06.669155 6296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:06.671083 6296 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:06.671162 6296 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.671343 6296 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:06.671396 6296 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:06.710268 6296 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:40061
I20260812 19:55:06.710376 6440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:40061 every 8 connection(s)
I20260812 19:55:06.711951 6296 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:55:06.721738 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6296
I20260812 19:55:06.721953 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:55:06.729108 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:06.731683 6443 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:55:06.732131 6443 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.733126 6443 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:55:06.733737 6442 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:06.733907 6442 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.734658 6442 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:55:06.735352 5966 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:40061)
I20260812 19:55:06.735868 5873 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:40061)
I20260812 19:55:06.737003 5873 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:37227
I20260812 19:55:06.737957 6441 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:55:06.738094 6441 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:06.738905 6441 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:55:06.740172 6065 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:40061)
W20260812 19:55:07.115731 6451 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:07.116120 6451 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:07.116199 6451 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:07.125293 6451 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:07.125478 6451 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:55:07.136230 6451 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:07.138025 6451 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:07.140092 6451 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:07.151376 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:07.151381 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:07.151769 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:07.152967 6451 server_base.cc:1061] running on GCE node
I20260812 19:55:07.153681 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:07.154762 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:07.155995 6451 hybrid_clock.cc:648] HybridClock initialized: now 1786564507155936 us; error 41 us; skew 500 ppm
I20260812 19:55:07.156383 6451 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:07.159170 6451 webserver.cc:533] Webserver started at http://127.25.14.3:36545/ using document root <none> and password file <none>
I20260812 19:55:07.160104 6451 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:07.160261 6451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:07.166569 6451 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:07.169584 6468 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:07.171172 6451 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:55:07.171375 6451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "3b85ff9460c543a18c7e6011d2c10fb4"
format_stamp: "Formatted at 2026-08-12 19:54:49 on dist-test-slave-p3t5"
I20260812 19:55:07.172211 6451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:07.204402 6451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:07.205390 6451 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:07.205783 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:07.206859 6451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:07.208751 6451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:07.208832 6451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:07.208968 6451 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:07.209022 6451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:07.228163 6287 heartbeater.cc:499] Master 127.25.14.62:38107 was elected leader, sending a full tablet report...
I20260812 19:55:07.248472 6451 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:41249
I20260812 19:55:07.248523 6594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:41249 every 8 connection(s)
I20260812 19:55:07.250272 6451 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:55:07.251916 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6451
I20260812 19:55:07.252216 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:55:07.270414 6598 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:55:07.270638 6597 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:07.270720 6596 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:55:07.270988 6596 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:07.270984 6598 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:07.270989 6597 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:07.272300 6596 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:55:07.272372 6597 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:55:07.272408 6598 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:55:07.274366 5873 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:41249)
I20260812 19:55:07.274672 5966 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:41249)
I20260812 19:55:07.274693 6065 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:41249)
I20260812 19:55:07.275542 5873 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:40905
I20260812 19:55:07.283181 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:07.291592 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:07.291858 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5838
I20260812 19:55:07.292868 5906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:07.405851 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5838
W20260812 19:55:07.435163 6048 connection.cc:573] server connection from 127.0.0.1:38440 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:55:07.436434 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--webserver_interface=127.25.14.62
--webserver_port=36573
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935 with env {}
I20260812 19:55:07.740108 6442 heartbeater.cc:499] Master 127.25.14.62:38107 was elected leader, sending a full tablet report...
W20260812 19:55:07.741992 6442 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:38107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:38107: connect: Connection refused (error 111)
W20260812 19:55:07.816763 6609 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:07.817201 6609 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:07.817349 6609 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:07.826095 6609 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:07.826246 6609 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:07.826412 6609 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:07.826581 6609 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:07.837666 6609 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=36573
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:07.839555 6609 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:07.841732 6609 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:07.851104 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:07.851137 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:07.852183 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:07.852631 6609 server_base.cc:1061] running on GCE node
I20260812 19:55:07.853471 6609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:07.855194 6609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:07.856416 6609 hybrid_clock.cc:648] HybridClock initialized: now 1786564507856368 us; error 34 us; skew 500 ppm
I20260812 19:55:07.856813 6609 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:07.859543 6609 webserver.cc:533] Webserver started at http://127.25.14.62:36573/ using document root <none> and password file <none>
I20260812 19:55:07.860425 6609 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:07.860561 6609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:07.866765 6609 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:07.870096 6627 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:07.871673 6609 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:55:07.871932 6609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14e6b1432b644b979baf2cd507c498d2"
format_stamp: "Formatted at 2026-08-12 19:54:46 on dist-test-slave-p3t5"
I20260812 19:55:07.872849 6609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:07.901674 6609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:07.902730 6609 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:07.903059 6609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:07.925649 6609 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38107
I20260812 19:55:07.925694 6688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38107 every 8 connection(s)
I20260812 19:55:07.927629 6609 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:07.935726 6689 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:07.936385 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6609
I20260812 19:55:07.936702 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6133
I20260812 19:55:07.939945 6689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap starting.
I20260812 19:55:07.947448 6280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:07.965806 6689 log.cc:826] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:07.978722 6689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:07.979463 6689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: Bootstrap complete.
I20260812 19:55:07.990257 6689 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:07.991329 6689 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e6b1432b644b979baf2cd507c498d2, State: Initialized, Role: FOLLOWER
I20260812 19:55:07.992312 6689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:07.993314 6696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:07.993760 6696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:07.994789 6689 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:08.013221 6710 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:08.013477 6710 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: loading cluster ID for follower catalog manager: success
I20260812 19:55:08.019078 6710 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: acquiring CA information for follower catalog manager: success
I20260812 19:55:08.022821 6710 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:08.060297 6609 master_runner.cc:264] No leader master found from master 14e6b1432b644b979baf2cd507c498d2
I20260812 19:55:08.060495 6609 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:08.083463 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6133
I20260812 19:55:08.109011 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=35919
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:08.278146 6597 heartbeater.cc:499] Master 127.25.14.62:38107 was elected leader, sending a full tablet report...
I20260812 19:55:08.290076 6648 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3b85ff9460c543a18c7e6011d2c10fb4" instance_seqno: 1786564507239544) as {username='slave'} at 127.25.14.3:40597; Asking this server to re-register.
I20260812 19:55:08.291216 6597 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:08.293138 6648 ts_manager.cc:194] Registered new tserver with Master: 3b85ff9460c543a18c7e6011d2c10fb4 (127.25.14.3:41249)
W20260812 19:55:08.542336 6719 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:08.542750 6719 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:08.542899 6719 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:08.552234 6719 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:08.552460 6719 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:08.566233 6726 raft_consensus.cc:493] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 14e6b1432b644b979baf2cd507c498d2)
I20260812 19:55:08.566461 6719 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35919
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:08.566478 6726 raft_consensus.cc:515] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:08.568303 6719 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:08.568490 6726 leader_election.cc:290] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107), 0003cb21d7ab4025aa0e3cddf7437aa7 (127.25.14.60:42935)
I20260812 19:55:08.571199 6719 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:55:08.576184 6658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c55730a891034fc1a14be12c5c4fcad5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "14e6b1432b644b979baf2cd507c498d2" is_pre_election: true
I20260812 19:55:08.576818 6658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c55730a891034fc1a14be12c5c4fcad5 in term 2.
I20260812 19:55:08.577589 5950 leader_election.cc:304] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e6b1432b644b979baf2cd507c498d2, c55730a891034fc1a14be12c5c4fcad5; no voters:
I20260812 19:55:08.578138 6726 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:55:08.578303 6726 raft_consensus.cc:493] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 14e6b1432b644b979baf2cd507c498d2)
I20260812 19:55:08.578420 6726 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:55:08.580056 6075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c55730a891034fc1a14be12c5c4fcad5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" is_pre_election: true
I20260812 19:55:08.581955 6726 raft_consensus.cc:515] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:08.582926 6726 leader_election.cc:290] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [CANDIDATE]: Term 3 election: Requested vote from peers 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107), 0003cb21d7ab4025aa0e3cddf7437aa7 (127.25.14.60:42935)
I20260812 19:55:08.583631 6075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c55730a891034fc1a14be12c5c4fcad5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7"
I20260812 19:55:08.583665 6658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c55730a891034fc1a14be12c5c4fcad5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "14e6b1432b644b979baf2cd507c498d2"
I20260812 19:55:08.583870 6658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:55:08.586597 6731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:08.588900 6658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c55730a891034fc1a14be12c5c4fcad5 in term 3.
W20260812 19:55:08.589646 6732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:08.589685 5950 leader_election.cc:304] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14e6b1432b644b979baf2cd507c498d2, c55730a891034fc1a14be12c5c4fcad5; no voters: 0003cb21d7ab4025aa0e3cddf7437aa7
I20260812 19:55:08.590106 6726 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:55:08.590734 6726 raft_consensus.cc:697] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [term 3 LEADER]: Becoming Leader. State: Replica: c55730a891034fc1a14be12c5c4fcad5, State: Running, Role: LEADER
W20260812 19:55:08.590986 6734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:08.591336 6726 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:08.591887 6719 server_base.cc:1061] running on GCE node
I20260812 19:55:08.592742 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:08.593641 6736 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55730a891034fc1a14be12c5c4fcad5. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:08.594117 6736 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:08.594913 6739 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:08.597177 6739 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:08.597957 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:08.598232 6739 catalog_manager.cc:1295] Loaded cluster ID: 735143c425ae4e9384bf7b208da5c653
I20260812 19:55:08.598337 6739 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:08.599205 6719 hybrid_clock.cc:648] HybridClock initialized: now 1786564508599124 us; error 87 us; skew 500 ppm
I20260812 19:55:08.599529 6719 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:08.599745 6739 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:08.600553 6739 catalog_manager.cc:6103] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5: Loaded TSK: 0
I20260812 19:55:08.601500 6739 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:55:08.602294 6719 webserver.cc:533] Webserver started at http://127.25.14.1:35919/ using document root <none> and password file <none>
I20260812 19:55:08.603134 6719 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:08.603268 6719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:08.609563 6719 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:55:08.612828 6743 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:08.614540 6719 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:08.614750 6719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:55:08.615604 6719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:08.643289 6719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:08.644358 6719 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:08.644694 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:08.645934 6719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:08.647850 6719 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:08.647926 6719 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:08.648102 6719 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:08.648176 6719 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:08.685111 6719 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35641
I20260812 19:55:08.685164 6869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35641 every 8 connection(s)
I20260812 19:55:08.686841 6719 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:08.694223 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6719
I20260812 19:55:08.706992 6871 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:08.707461 6871 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:08.708566 6870 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:55:08.708702 6870 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:08.708674 6871 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:55:08.707046 6872 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:55:08.708902 6872 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:08.709394 6870 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:55:08.709523 6872 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:55:08.710906 6648 ts_manager.cc:194] Registered new tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:08.711119 6065 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:08.712167 5966 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:08.714143 5966 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:33435
I20260812 19:55:08.754227 6442 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:08.754570 6442 heartbeater.cc:499] Master 127.25.14.62:38107 was elected leader, sending a full tablet report...
I20260812 19:55:08.755843 6648 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "36caa9630e634111bdc132b24dcec0fb" instance_seqno: 1786564506700596) as {username='slave'} at 127.25.14.2:58989; Asking this server to re-register.
I20260812 19:55:08.756690 6442 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:08.758139 6648 ts_manager.cc:194] Registered new tserver with Master: 36caa9630e634111bdc132b24dcec0fb (127.25.14.2:40061)
I20260812 19:55:08.829730 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6719
I20260812 19:55:08.849843 6864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:08.907128 6886 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 14e6b1432b644b979baf2cd507c498d2)
I20260812 19:55:08.907573 6886 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } }
I20260812 19:55:08.910212 6886 leader_election.cc:290] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14e6b1432b644b979baf2cd507c498d2 (127.25.14.62:38107), c55730a891034fc1a14be12c5c4fcad5 (127.25.14.61:39035)
I20260812 19:55:08.911844 5978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c55730a891034fc1a14be12c5c4fcad5" is_pre_election: true
I20260812 19:55:08.920399 6658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "14e6b1432b644b979baf2cd507c498d2" is_pre_election: true
I20260812 19:55:08.920851 6658 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0003cb21d7ab4025aa0e3cddf7437aa7 in current term 3: Already voted for candidate c55730a891034fc1a14be12c5c4fcad5 in this term.
I20260812 19:55:08.921777 6048 leader_election.cc:304] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0003cb21d7ab4025aa0e3cddf7437aa7; no voters: 14e6b1432b644b979baf2cd507c498d2, c55730a891034fc1a14be12c5c4fcad5
I20260812 19:55:08.922278 6886 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:55:08.926502 6886 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:55:08.943971 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6719
I20260812 19:55:08.969518 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=35919
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260812 19:55:08.990967 6075 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [term 3 FOLLOWER]: Refusing update from remote peer c55730a891034fc1a14be12c5c4fcad5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:55:08.991989 6736 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:55:08.994513 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55730a891034fc1a14be12c5c4fcad5. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:08.994867 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:08.997207 6726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:08.997550 6726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:08.998672 6658 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [term 3 FOLLOWER]: Refusing update from remote peer c55730a891034fc1a14be12c5c4fcad5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:55:08.999840 6726 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:55:09.006431 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:09.006759 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0003cb21d7ab4025aa0e3cddf7437aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:09.008116 6897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:09.008581 6897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:09.009270 6894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55730a891034fc1a14be12c5c4fcad5. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:09.009588 6894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6b1432b644b979baf2cd507c498d2 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:55:09.355350 6888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:09.355705 6888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:09.355795 6888 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:55:09.355861 6888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:09.366114 6888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:09.366371 6888 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:09.377058 6888 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35919
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:09.379051 6888 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:09.381170 6888 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:09.392380 6905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:09.392411 6904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:09.392643 6907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:09.394207 6888 server_base.cc:1061] running on GCE node
I20260812 19:55:09.394861 6888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:09.395912 6888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:09.397173 6888 hybrid_clock.cc:648] HybridClock initialized: now 1786564509397076 us; error 69 us; skew 500 ppm
I20260812 19:55:09.397498 6888 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:09.399930 6888 webserver.cc:533] Webserver started at http://127.25.14.1:35919/ using document root <none> and password file <none>
I20260812 19:55:09.400748 6888 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:09.400933 6888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:09.407410 6888 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:09.410478 6914 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:09.411998 6888 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:09.412190 6888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:55:09.413021 6888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:09.455706 6888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:09.456724 6888 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:09.457039 6888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:09.458359 6888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:09.458756 6726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c55730a891034fc1a14be12c5c4fcad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e6b1432b644b979baf2cd507c498d2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38107 } } peers { permanent_uuid: "c55730a891034fc1a14be12c5c4fcad5" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 39035 } } peers { permanent_uuid: "0003cb21d7ab4025aa0e3cddf7437aa7" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42935 } } }
I20260812 19:55:09.459002 6726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55730a891034fc1a14be12c5c4fcad5 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:09.460564 6888 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:09.460681 6888 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:09.460814 6888 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:09.460909 6888 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:55:09.461314 6888 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:55:09.461423 6888 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:55:09.461491 6888 fault_injection.cc:59] injecting fault for kudu (pid 6888): FLAGS_fault_before_start (process will exit)
I20260812 19:55:09.490263 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--webserver_port=35919
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:09.741856 6443 heartbeater.cc:499] Master 127.25.14.61:39035 was elected leader, sending a full tablet report...
W20260812 19:55:09.881304 6925 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:09.881749 6925 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:09.881848 6925 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:09.890645 6925 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:09.890831 6925 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:09.901661 6925 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35919
--tserver_master_addrs=127.25.14.62:38107,127.25.14.61:39035,127.25.14.60:42935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:09.903515 6925 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:09.905493 6925 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:09.916939 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:09.916939 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:09.917193 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:09.917881 6925 server_base.cc:1061] running on GCE node
I20260812 19:55:09.918740 6925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:09.919880 6925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:09.921150 6925 hybrid_clock.cc:648] HybridClock initialized: now 1786564509921095 us; error 59 us; skew 500 ppm
I20260812 19:55:09.921518 6925 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:09.923964 6925 webserver.cc:533] Webserver started at http://127.25.14.1:35919/ using document root <none> and password file <none>
I20260812 19:55:09.924835 6925 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:09.925001 6925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:09.931668 6925 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:55:09.935034 6940 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:09.936697 6925 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:55:09.936932 6925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9cfb1e80b09c4b60b2d5d703971ee29f"
format_stamp: "Formatted at 2026-08-12 19:54:48 on dist-test-slave-p3t5"
I20260812 19:55:09.937839 6925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:09.958490 6925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:09.959689 6925 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:09.960016 6925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:09.961037 6925 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:09.962996 6925 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:09.963102 6925 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:09.963253 6925 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:09.963323 6925 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:10.000669 6925 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35641
I20260812 19:55:10.000715 7067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35641 every 8 connection(s)
I20260812 19:55:10.002501 6925 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:10.011068 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6925
I20260812 19:55:10.011528 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6925
I20260812 19:55:10.037477 7068 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42935
I20260812 19:55:10.038105 7068 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:10.037504 7069 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38107
I20260812 19:55:10.038304 7069 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:10.039477 7069 heartbeater.cc:507] Master 127.25.14.62:38107 requested a full tablet report, sending...
I20260812 19:55:10.041047 7068 heartbeater.cc:507] Master 127.25.14.60:42935 requested a full tablet report, sending...
I20260812 19:55:10.042331 6065 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:10.043005 6648 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:10.043133 7062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.134172 7071 heartbeater.cc:344] Connected to a master server at 127.25.14.61:39035
I20260812 19:55:10.134475 7071 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:10.135159 7071 heartbeater.cc:507] Master 127.25.14.61:39035 requested a full tablet report, sending...
I20260812 19:55:10.135726 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6925
I20260812 19:55:10.139537 5966 ts_manager.cc:194] Re-registered known tserver with Master: 9cfb1e80b09c4b60b2d5d703971ee29f (127.25.14.1:35641)
I20260812 19:55:10.140601 5966 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:43505
W20260812 19:55:10.163681 5950 connection.cc:573] server connection from 127.25.14.1:43505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:55:10.164013 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6296
I20260812 19:55:10.173095 6436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.262902 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6296
I20260812 19:55:10.280994 6598 heartbeater.cc:499] Master 127.25.14.61:39035 was elected leader, sending a full tablet report...
I20260812 19:55:10.287958 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6451
I20260812 19:55:10.296797 6590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.387679 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6451
I20260812 19:55:10.413895 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6609
I20260812 19:55:10.421025 6682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.529739 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6609
I20260812 19:55:10.557799 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5913
I20260812 19:55:10.559512 6000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.678614 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5913
I20260812 19:55:10.713037 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6007
I20260812 19:55:10.714576 6100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:10.851109 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6007
Shutting down hive metastore.
2026-08-12T19:55:11Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/4 (41515 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/5
2026-08-12T19:55:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:11Z Disabled control of system clock
2026-08-12T19:55:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:11Z Disabled control of system clock
2026-08-12T19:55:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:11Z Disabled control of system clock
2026-08-12T19:55:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:11Z Disabled control of system clock
2026-08-12T19:55:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:11Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:55:19: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:55:28.209160 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
W20260812 19:55:28.583837 7426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:28.584270 7426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:28.584426 7426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:28.593041 7426 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:28.593197 7426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:28.593307 7426 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:28.593393 7426 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:28.604063 7426 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:28.606078 7426 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:28.608175 7426 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:28.618050 7432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:28.618121 7431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:28.619098 7434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:28.619393 7426 server_base.cc:1061] running on GCE node
I20260812 19:55:28.620296 7426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:28.622438 7426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:28.623800 7426 hybrid_clock.cc:648] HybridClock initialized: now 1786564528623648 us; error 117 us; skew 500 ppm
I20260812 19:55:28.624238 7426 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:28.626786 7426 webserver.cc:533] Webserver started at http://127.25.14.62:40147/ using document root <none> and password file <none>
I20260812 19:55:28.627727 7426 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:28.627882 7426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:28.628286 7426 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:28.630951 7426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:28.631789 7426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:28.637945 7426 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:55:28.641733 7441 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:28.643195 7426 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:28.643400 7426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:28.643677 7426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:28.660802 7426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:28.661933 7426 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:28.662243 7426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:28.684713 7426 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38251
I20260812 19:55:28.684734 7498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38251 every 8 connection(s)
I20260812 19:55:28.686537 7426 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:28.688974 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7426
I20260812 19:55:28.689216 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:55:28.690699 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46699
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:28.692744 7499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:55:28.702190 7499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:28.725907 7499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:28.727309 7444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
W20260812 19:55:28.729488 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
I20260812 19:55:28.778935 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } attempt: 1
W20260812 19:55:28.781437 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
I20260812 19:55:28.849746 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } attempt: 2
W20260812 19:55:28.852180 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
I20260812 19:55:28.943522 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } attempt: 3
W20260812 19:55:28.945842 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
W20260812 19:55:29.071969 7501 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:29.072376 7501 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:29.072515 7501 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:29.081179 7501 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:29.081292 7501 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:29.081400 7501 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:29.081481 7501 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:29.089169 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } attempt: 4
W20260812 19:55:29.091557 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111)
I20260812 19:55:29.092696 7501 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:29.094542 7501 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:29.096465 7501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:29.105688 7511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:29.105782 7512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:29.106472 7514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:29.107178 7501 server_base.cc:1061] running on GCE node
I20260812 19:55:29.108035 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:29.109935 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:29.111173 7501 hybrid_clock.cc:648] HybridClock initialized: now 1786564529111026 us; error 113 us; skew 500 ppm
I20260812 19:55:29.111553 7501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:29.114010 7501 webserver.cc:533] Webserver started at http://127.25.14.61:39307/ using document root <none> and password file <none>
I20260812 19:55:29.114828 7501 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:29.114960 7501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:29.115352 7501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:29.120002 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "a476e34d821e4ba7a38723a9919b5b68"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.120882 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "a476e34d821e4ba7a38723a9919b5b68"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.127043 7501 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 19:55:29.130849 7521 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:29.132447 7501 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:55:29.132654 7501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "a476e34d821e4ba7a38723a9919b5b68"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.132931 7501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:29.149704 7501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:29.150676 7501 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:29.151000 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:29.172931 7501 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46699
I20260812 19:55:29.172958 7580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46699 every 8 connection(s)
I20260812 19:55:29.174679 7501 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:55:29.179418 7582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:55:29.182802 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7501
I20260812 19:55:29.183040 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:55:29.184563 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39357
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:29.188519 7582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:29.204310 7582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:29.216611 7582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:29.217963 7525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
W20260812 19:55:29.219846 7582 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.269258 7582 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 1
W20260812 19:55:29.271606 7582 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.339920 7582 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 2
W20260812 19:55:29.342327 7582 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.390920 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } attempt: 5
I20260812 19:55:29.398538 7499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:29.400774 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.433650 7582 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 3
W20260812 19:55:29.435989 7582 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.452082 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 1
W20260812 19:55:29.455132 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.523420 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 2
W20260812 19:55:29.525782 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
W20260812 19:55:29.570899 7585 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:29.571337 7585 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:29.571481 7585 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:55:29.579312 7582 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 4
W20260812 19:55:29.580499 7585 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:29.580631 7585 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:29.580760 7585 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:29.580843 7585 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 19:55:29.581560 7582 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.591724 7585 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:29.593580 7585 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:29.595587 7585 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:29.605087 7595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:29.605077 7594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:29.605366 7597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:29.606479 7585 server_base.cc:1061] running on GCE node
I20260812 19:55:29.607451 7585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:29.609321 7585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:29.610761 7585 hybrid_clock.cc:648] HybridClock initialized: now 1786564529610577 us; error 140 us; skew 500 ppm
I20260812 19:55:29.611232 7585 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:29.613576 7585 webserver.cc:533] Webserver started at http://127.25.14.60:43643/ using document root <none> and password file <none>
I20260812 19:55:29.614528 7585 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:29.614663 7585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:29.615118 7585 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:29.618144 7585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.619236 7585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.625357 7585 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 19:55:29.629323 7605 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:29.630995 7585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:29.631203 7585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:29.631474 7585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:29.632012 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 3
W20260812 19:55:29.634243 7499 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111)
I20260812 19:55:29.660130 7585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:29.661142 7585 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:29.661473 7585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:29.683238 7585 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:39357
I20260812 19:55:29.683259 7661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:39357 every 8 connection(s)
I20260812 19:55:29.685137 7585 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:55:29.687788 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7585
I20260812 19:55:29.688033 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:55:29.691370 7662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:55:29.700423 7662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:29.715240 7662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:29.722004 7662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:55:29.732120 7662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Bootstrap starting.
I20260812 19:55:29.735972 7662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:29.737254 7662 log.cc:826] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:29.740118 7662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: No bootstrap required, opened a new log
I20260812 19:55:29.746142 7662 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.746521 7662 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:29.746670 7662 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdcf1ae0c8c45da8e8af22a4962ab4e, State: Initialized, Role: FOLLOWER
I20260812 19:55:29.747455 7662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.748684 7668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:29.749027 7668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:29.749784 7662 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:55:29.760221 7680 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:29.760387 7680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:55:29.757884 7585 master_runner.cc:187] Error getting master registration for 127.25.14.62:38251: OK, instance_id { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" instance_seqno: 1786564528678698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:55:29.765166 7585 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:29.811503 7499 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 4
I20260812 19:55:29.825316 7499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap starting.
I20260812 19:55:29.830015 7499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:29.831609 7499 log.cc:826] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:29.834918 7499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: No bootstrap required, opened a new log
I20260812 19:55:29.842996 7499 raft_consensus.cc:359] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.843385 7499 raft_consensus.cc:385] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:29.843531 7499 raft_consensus.cc:740] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Initialized, Role: FOLLOWER
I20260812 19:55:29.844321 7499 consensus_queue.cc:260] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.846264 7683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:29.846560 7683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:29.847656 7499 sys_catalog.cc:565] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:55:29.858687 7697 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:29.858884 7697 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:55:29.858228 7426 master_runner.cc:187] Error getting master registration for 127.25.14.61:46699: OK, instance_id { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" instance_seqno: 1786564529167061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:55:29.866204 7426 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:29.880847 7582 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } attempt: 5
I20260812 19:55:29.895267 7582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Bootstrap starting.
I20260812 19:55:29.899984 7582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:29.901539 7582 log.cc:826] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:29.910058 7582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: No bootstrap required, opened a new log
W20260812 19:55:29.913889 7613 tablet.cc:2406] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 19:55:29.915344 7447 tablet.cc:2406] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:55:29.918725 7582 raft_consensus.cc:359] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.919168 7582 raft_consensus.cc:385] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:29.919373 7582 raft_consensus.cc:740] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a476e34d821e4ba7a38723a9919b5b68, State: Initialized, Role: FOLLOWER
I20260812 19:55:29.920254 7582 consensus_queue.cc:260] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:29.922032 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:29.922436 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:29.924692 7582 sys_catalog.cc:565] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:29.960609 7501 master_runner.cc:264] No leader master found from master a476e34d821e4ba7a38723a9919b5b68
I20260812 19:55:29.960788 7501 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:29.964634 7713 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:29.964959 7713 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:55:30.020066 7668 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:55:30.020419 7668 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:30.022607 7668 leader_election.cc:290] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699)
I20260812 19:55:30.023787 7683 raft_consensus.cc:493] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:55:30.024067 7683 raft_consensus.cc:515] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:30.025856 7683 leader_election.cc:290] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:30.027555 7472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d" is_pre_election: true
I20260812 19:55:30.028019 7472 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in term 0.
I20260812 19:55:30.028816 7551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68" is_pre_election: true
I20260812 19:55:30.029408 7551 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in term 0.
I20260812 19:55:30.030053 7550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68" is_pre_election: true
I20260812 19:55:30.030422 7550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 0.
I20260812 19:55:30.031561 7634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" is_pre_election: true
I20260812 19:55:30.031997 7634 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 0.
I20260812 19:55:30.032794 7444 leader_election.cc:304] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters:
I20260812 19:55:30.034287 7607 leader_election.cc:304] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e, a020a9db64854326bffffac643bbfa5d; no voters:
I20260812 19:55:30.035504 7683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:55:30.035667 7683 raft_consensus.cc:493] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:55:30.035764 7683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:30.037760 7668 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:55:30.037923 7668 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:55:30.038034 7668 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:30.048096 7668 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:30.049005 7683 raft_consensus.cc:515] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:30.053157 7551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68"
I20260812 19:55:30.053452 7551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:30.057215 7551 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 1.
I20260812 19:55:30.058189 7444 leader_election.cc:304] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters:
I20260812 19:55:30.065507 7472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d"
I20260812 19:55:30.066048 7472 raft_consensus.cc:2393] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in current term 1: Already voted for candidate a020a9db64854326bffffac643bbfa5d in this term.
I20260812 19:55:30.068131 7634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
I20260812 19:55:30.068506 7634 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a020a9db64854326bffffac643bbfa5d in current term 1: Already voted for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in this term.
I20260812 19:55:30.069746 7683 leader_election.cc:290] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 1 election: Requested vote from peers a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:30.070884 7551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68"
I20260812 19:55:30.071388 7551 raft_consensus.cc:2393] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in current term 1: Already voted for candidate a020a9db64854326bffffac643bbfa5d in this term.
I20260812 19:55:30.072150 7608 leader_election.cc:304] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e; no voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68
I20260812 19:55:30.079006 7683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:55:30.079679 7668 leader_election.cc:290] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 1 election: Requested vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699)
I20260812 19:55:30.080384 7668 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:55:30.080700 7683 raft_consensus.cc:697] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 LEADER]: Becoming Leader. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Running, Role: LEADER
I20260812 19:55:30.082645 7683 consensus_queue.cc:237] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:30.100705 7715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.101277 7715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
W20260812 19:55:30.102002 7444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.14.61:46699, user_credentials={real_user=slave}} blocked reactor thread for 43874.2us
I20260812 19:55:30.104035 7719 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:30.106827 7719 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:30.124058 7634 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Refusing update from remote peer a020a9db64854326bffffac643bbfa5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:55:30.125325 7715 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:55:30.136950 7551 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Refusing update from remote peer a020a9db64854326bffffac643bbfa5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:55:30.138361 7715 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 19:55:30.153970 7527 tablet.cc:2406] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:55:30.168056 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.168466 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:30.208335 7683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.208714 7683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:30.217373 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.217715 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:30.218336 7719 catalog_manager.cc:1383] Generated new cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:30.218468 7719 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:30.218713 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.218992 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:30.232044 7715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.232317 7715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:30.236686 7722 mvcc.cc:204] Tried to move back new op lower bound from 7317768315372298240 to 7317768315238764544. Current Snapshot: MvccSnapshot[applied={T|T < 7317768315372298240}]
I20260812 19:55:30.255283 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:30.255533 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:30.298043 7719 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:55:30.300010 7719 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:30.316058 7719 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Generated new TSK 0
I20260812 19:55:30.317934 7719 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:55:30.354753 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:55:30.749454 7728 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:30.749909 7728 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:30.750053 7728 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:30.758977 7728 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:30.759232 7728 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:30.762564 7680 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:30.762662 7680 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: loading cluster ID for follower catalog manager: success
I20260812 19:55:30.765690 7680 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: acquiring CA information for follower catalog manager: success
I20260812 19:55:30.767825 7680 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:30.770236 7728 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:30.771965 7728 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:30.774045 7728 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:30.784562 7735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:30.784504 7733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:30.785321 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:30.786144 7728 server_base.cc:1061] running on GCE node
I20260812 19:55:30.786886 7728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:30.788139 7728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:30.789472 7728 hybrid_clock.cc:648] HybridClock initialized: now 1786564530789304 us; error 126 us; skew 500 ppm
I20260812 19:55:30.789875 7728 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:30.792338 7728 webserver.cc:533] Webserver started at http://127.25.14.1:45877/ using document root <none> and password file <none>
I20260812 19:55:30.793195 7728 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:30.793337 7728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:30.793799 7728 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:30.796439 7728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:30.797391 7728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:30.803936 7728 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260812 19:55:30.807963 7746 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:30.809607 7728 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:55:30.809841 7728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:30.810124 7728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:30.830870 7728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:30.831897 7728 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:30.832230 7728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:30.833287 7728 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:30.835261 7728 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:30.835430 7728 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:30.835562 7728 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:30.835659 7728 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:30.873405 7728 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43487
I20260812 19:55:30.873471 7886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43487 every 8 connection(s)
I20260812 19:55:30.875023 7728 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:30.876104 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7728
I20260812 19:55:30.876327 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:55:30.882254 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:30.928922 7887 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:30.929325 7887 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:30.929698 7888 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:30.929955 7888 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:30.931138 7888 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:30.931267 7887 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:30.932482 7890 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:30.932643 7890 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:30.933140 7890 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:30.933801 7460 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:30.934131 7624 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:30.934926 7540 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:30.936046 7460 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:59635
I20260812 19:55:30.967173 7713 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:30.967288 7713 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: loading cluster ID for follower catalog manager: success
I20260812 19:55:30.970134 7713 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: acquiring CA information for follower catalog manager: success
I20260812 19:55:30.972054 7713 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:55:31.307796 7899 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:31.308192 7899 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:31.308343 7899 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:31.317751 7899 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:31.318027 7899 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:55:31.328481 7899 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:31.330262 7899 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:31.332199 7899 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:31.343395 7906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:31.343598 7908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:31.343418 7905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:31.345157 7899 server_base.cc:1061] running on GCE node
I20260812 19:55:31.345798 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:31.346959 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:31.348290 7899 hybrid_clock.cc:648] HybridClock initialized: now 1786564531348131 us; error 124 us; skew 500 ppm
I20260812 19:55:31.348721 7899 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:31.351253 7899 webserver.cc:533] Webserver started at http://127.25.14.2:36459/ using document root <none> and password file <none>
I20260812 19:55:31.352108 7899 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:31.352246 7899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:31.352649 7899 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:31.355427 7899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "1cb884363ca343429823b38dbcfc1e1f"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.356295 7899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "1cb884363ca343429823b38dbcfc1e1f"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.362154 7899 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 19:55:31.366407 7916 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.368283 7899 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:31.368512 7899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1cb884363ca343429823b38dbcfc1e1f"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.368744 7899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:31.384816 7899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:31.385895 7899 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:31.386204 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:31.387295 7899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:31.389180 7899 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:31.389293 7899 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.389441 7899 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:31.389518 7899 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.426122 7899 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:34963
I20260812 19:55:31.426189 8061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:34963 every 8 connection(s)
I20260812 19:55:31.428210 7899 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:55:31.431524 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7899
I20260812 19:55:31.431759 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:55:31.437438 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:31.465710 8064 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:31.466197 8064 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:31.467433 8064 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:31.469156 8063 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:31.469296 8063 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:31.469584 7460 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:34963)
I20260812 19:55:31.469887 8063 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:31.470851 7460 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:57033
I20260812 19:55:31.470921 7624 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:34963)
I20260812 19:55:31.471338 8065 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:31.471524 8065 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:31.472097 8065 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:31.473310 7540 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:34963)
W20260812 19:55:31.833050 8075 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:31.833485 8075 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:31.833674 8075 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:31.842669 8075 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:31.842917 8075 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:55:31.854117 8075 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:31.856021 8075 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:31.858238 8075 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:31.869261 8080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:31.869261 8081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:31.869721 8083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:31.870237 8075 server_base.cc:1061] running on GCE node
I20260812 19:55:31.870920 8075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:31.872153 8075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:31.873648 8075 hybrid_clock.cc:648] HybridClock initialized: now 1786564531873439 us; error 144 us; skew 500 ppm
I20260812 19:55:31.874086 8075 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:31.877032 8075 webserver.cc:533] Webserver started at http://127.25.14.3:42275/ using document root <none> and password file <none>
I20260812 19:55:31.878149 8075 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:31.878279 8075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:31.878661 8075 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:31.881742 8075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "51c635028a6f41de8592e229bcdf3fa6"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.882560 8075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "51c635028a6f41de8592e229bcdf3fa6"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.888594 8075 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:55:31.892580 8090 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.894297 8075 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:31.894536 8075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "51c635028a6f41de8592e229bcdf3fa6"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:31.894798 8075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:31.939292 7888 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:31.939951 8075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:31.941223 8075 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:31.941550 8075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:31.942653 8075 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:31.944736 8075 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:31.944885 8075 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.945031 8075 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:31.945130 8075 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:31.982250 8075 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:38025
I20260812 19:55:31.982352 8220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:38025 every 8 connection(s)
I20260812 19:55:31.983939 8075 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:55:31.992401 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8075
I20260812 19:55:31.992666 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:55:32.017401 8226 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:32.017762 8224 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:32.017926 8226 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:32.017961 8224 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:32.018242 8222 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:32.018378 8222 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:32.019101 8222 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:32.019124 8226 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:32.019825 8224 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:32.021085 7460 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38025)
I20260812 19:55:32.021204 7540 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38025)
I20260812 19:55:32.021513 7624 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38025)
I20260812 19:55:32.022132 7460 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:43605
I20260812 19:55:32.031544 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:32.038987 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:32.039218 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7426
I20260812 19:55:32.040220 7492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:32.153247 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7426
I20260812 19:55:32.187144 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--webserver_interface=127.25.14.62
--webserver_port=40147
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:32.475044 8064 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
W20260812 19:55:32.476728 8064 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:38251 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:38251: connect: Connection refused (error 111)
W20260812 19:55:32.574550 8236 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:32.574990 8236 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:32.575078 8236 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:32.584513 8236 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:32.584631 8236 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:32.584690 8236 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:32.584733 8236 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:32.595772 8236 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=40147
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:32.597564 8236 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:32.599638 8236 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:32.609185 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:32.609256 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:32.610172 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:32.610587 8236 server_base.cc:1061] running on GCE node
I20260812 19:55:32.611433 8236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:32.613355 8236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:32.616719 8236 hybrid_clock.cc:648] HybridClock initialized: now 1786564532616535 us; error 141 us; skew 500 ppm
I20260812 19:55:32.617189 8236 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:32.619915 8236 webserver.cc:533] Webserver started at http://127.25.14.62:40147/ using document root <none> and password file <none>
I20260812 19:55:32.620793 8236 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:32.620967 8236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:32.627184 8236 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:32.630874 8255 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:32.632622 8236 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:32.632938 8236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:32.633944 8236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:32.659106 8236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:32.660245 8236 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:32.660616 8236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:32.685465 8236 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38251
I20260812 19:55:32.685470 8319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38251 every 8 connection(s)
I20260812 19:55:32.687603 8236 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:32.688750 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8236
I20260812 19:55:32.689098 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7728
I20260812 19:55:32.701152 8320 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:32.705575 8320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap starting.
I20260812 19:55:32.706063 7880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:32.733667 8320 log.cc:826] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:32.745541 8320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:32.746250 8320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap complete.
I20260812 19:55:32.757246 8320 raft_consensus.cc:359] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:32.758364 8320 raft_consensus.cc:740] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Initialized, Role: FOLLOWER
I20260812 19:55:32.759335 8320 consensus_queue.cc:260] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:32.760315 8327 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:32.760612 8327 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:32.761737 8320 sys_catalog.cc:565] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:32.773002 8341 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:32.773141 8341 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: loading cluster ID for follower catalog manager: success
I20260812 19:55:32.777947 8341 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: acquiring CA information for follower catalog manager: success
I20260812 19:55:32.781019 8341 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:32.823809 8236 master_runner.cc:264] No leader master found from master a020a9db64854326bffffac643bbfa5d
I20260812 19:55:32.824020 8236 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:32.831017 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7728
I20260812 19:55:32.856604 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43487
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=45877
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:33.024823 8224 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:33.036793 8276 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "51c635028a6f41de8592e229bcdf3fa6" instance_seqno: 1786564531973706) as {username='slave'} at 127.25.14.3:39155; Asking this server to re-register.
I20260812 19:55:33.038156 8224 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:33.040520 8276 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38025)
W20260812 19:55:33.248514 8347 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:33.248852 8347 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:33.248930 8347 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:33.257822 8347 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:33.257993 8347 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:33.269062 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45877
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:33.271075 8347 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:33.272970 8347 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:33.283500 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:33.284682 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:33.285531 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:33.286451 8347 server_base.cc:1061] running on GCE node
I20260812 19:55:33.287132 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:33.288431 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:33.289721 8347 hybrid_clock.cc:648] HybridClock initialized: now 1786564533289584 us; error 102 us; skew 500 ppm
I20260812 19:55:33.290086 8347 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:33.292481 8347 webserver.cc:533] Webserver started at http://127.25.14.1:45877/ using document root <none> and password file <none>
I20260812 19:55:33.293390 8347 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:33.293563 8347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:33.299896 8347 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:55:33.303308 8369 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:33.304932 8347 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:55:33.305158 8347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:33.305958 8370 raft_consensus.cc:493] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:33.306092 8347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:33.306206 8370 raft_consensus.cc:515] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:33.307626 8370 leader_election.cc:290] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:33.308656 7633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" is_pre_election: true
I20260812 19:55:33.315405 8287 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d" is_pre_election: true
I20260812 19:55:33.316051 8287 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a476e34d821e4ba7a38723a9919b5b68 in term 1.
I20260812 19:55:33.316740 7523 leader_election.cc:304] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e
I20260812 19:55:33.317229 8370 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:55:33.317386 8370 raft_consensus.cc:493] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:33.317518 8370 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:33.319960 8370 raft_consensus.cc:515] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:33.320737 8370 leader_election.cc:290] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 2 election: Requested vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:33.321370 8287 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d"
I20260812 19:55:33.321430 7633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
I20260812 19:55:33.321676 8287 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:33.325729 8287 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a476e34d821e4ba7a38723a9919b5b68 in term 2.
I20260812 19:55:33.326269 7523 leader_election.cc:304] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e
I20260812 19:55:33.326638 8370 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:55:33.327158 8370 raft_consensus.cc:697] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 LEADER]: Becoming Leader. State: Replica: a476e34d821e4ba7a38723a9919b5b68, State: Running, Role: LEADER
I20260812 19:55:33.327663 8370 consensus_queue.cc:237] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:33.329648 8376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a476e34d821e4ba7a38723a9919b5b68. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.329959 8376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:33.330724 8378 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:33.332554 8378 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:33.333371 8378 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:33.333478 8378 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:33.334673 8378 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:33.335500 8378 catalog_manager.cc:6103] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Loaded TSK: 0
I20260812 19:55:33.336433 8378 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:55:33.342729 8347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:33.343694 8347 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:33.344012 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:33.345047 8347 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:33.346915 8347 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:33.347024 8347 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:33.347183 8347 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:33.347271 8347 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:33.384528 8347 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43487
I20260812 19:55:33.384564 8503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43487 every 8 connection(s)
I20260812 19:55:33.386317 8347 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:33.388911 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8347
I20260812 19:55:33.456467 8505 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:33.456647 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8347
I20260812 19:55:33.456959 8505 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:33.457206 8504 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:33.457362 8504 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:33.458423 8505 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:33.459133 8504 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:33.461158 7624 ts_manager.cc:194] Re-registered known tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:33.473392 8506 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:33.473654 8506 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:33.474344 8506 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:33.475744 7540 ts_manager.cc:194] Re-registered known tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:33.478725 7540 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:54639
I20260812 19:55:33.490195 8276 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:43487)
I20260812 19:55:33.495414 8064 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:33.496199 8064 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:33.497453 8276 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1cb884363ca343429823b38dbcfc1e1f" instance_seqno: 1786564531417839) as {username='slave'} at 127.25.14.2:58105; Asking this server to re-register.
I20260812 19:55:33.498431 8064 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:33.499804 8276 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:34963)
I20260812 19:55:33.500748 8499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:33.504468 8520 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:33.504808 8520 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:33.512521 8520 leader_election.cc:290] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699)
I20260812 19:55:33.513504 7550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68" is_pre_election: true
I20260812 19:55:33.522778 8287 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d" is_pre_election: true
I20260812 19:55:33.523236 8287 raft_consensus.cc:2393] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in current term 2: Already voted for candidate a476e34d821e4ba7a38723a9919b5b68 in this term.
I20260812 19:55:33.524101 7607 leader_election.cc:304] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e; no voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68
I20260812 19:55:33.524520 8520 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:55:33.528631 8520 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:55:33.602808 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8347
I20260812 19:55:33.628518 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43487
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=45877
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260812 19:55:33.879364 7633 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Refusing update from remote peer a476e34d821e4ba7a38723a9919b5b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:55:33.880352 8524 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:55:33.882572 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a476e34d821e4ba7a38723a9919b5b68. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.883015 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:33.885105 8526 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.885483 8526 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:33.886804 8287 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Refusing update from remote peer a476e34d821e4ba7a38723a9919b5b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:55:33.894508 8526 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:55:33.895236 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.895505 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:33.899362 8536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.899722 8536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:33.900391 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a476e34d821e4ba7a38723a9919b5b68. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:33.900744 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:55:34.011879 8523 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:34.012302 8523 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:34.012455 8523 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:55:34.012562 8523 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:34.021531 8523 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:34.021840 8523 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:34.033478 8523 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45877
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:34.035379 8523 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:34.037456 8523 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:34.048966 8542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:34.048959 8543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:34.049939 8546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:34.050714 8523 server_base.cc:1061] running on GCE node
I20260812 19:55:34.051443 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:34.052752 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:34.056129 8523 hybrid_clock.cc:648] HybridClock initialized: now 1786564534055948 us; error 117 us; skew 500 ppm
I20260812 19:55:34.056516 8523 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:34.059037 8523 webserver.cc:533] Webserver started at http://127.25.14.1:45877/ using document root <none> and password file <none>
I20260812 19:55:34.059937 8523 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:34.060112 8523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:34.066773 8523 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:34.070124 8556 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:34.071767 8523 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:55:34.072021 8523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:34.072826 8523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:34.118762 8523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:34.119824 8523 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:34.120164 8523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:34.121342 8523 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:34.123343 8523 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:34.123481 8523 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:34.123627 8523 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:34.123729 8523 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:55:34.124140 8523 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:55:34.124248 8523 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:55:34.124317 8523 fault_injection.cc:59] injecting fault for kudu (pid 8523): FLAGS_fault_before_start (process will exit)
I20260812 19:55:34.149497 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43487
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=45877
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:34.478209 8065 heartbeater.cc:499] Master 127.25.14.61:46699 was elected leader, sending a full tablet report...
I20260812 19:55:34.482312 8565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a476e34d821e4ba7a38723a9919b5b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:34.482611 8565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: LEADER
W20260812 19:55:34.524987 8564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:34.525343 8564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:34.525422 8564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:34.534245 8564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:34.534431 8564 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:34.545185 8564 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:43627,127.25.14.21:40721,127.25.14.22:41479,127.25.14.23:44839,127.25.14.24:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45877
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:34.546916 8564 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:34.548863 8564 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:34.560743 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:34.561470 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:34.562372 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:34.563251 8564 server_base.cc:1061] running on GCE node
I20260812 19:55:34.564402 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:34.566172 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:34.569548 8564 hybrid_clock.cc:648] HybridClock initialized: now 1786564534569340 us; error 161 us; skew 500 ppm
I20260812 19:55:34.570019 8564 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:34.572412 8564 webserver.cc:533] Webserver started at http://127.25.14.1:45877/ using document root <none> and password file <none>
I20260812 19:55:34.573228 8564 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:34.573330 8564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:34.579618 8564 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:34.583324 8585 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:34.585072 8564 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:34.585248 8564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:34.586115 8564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:34.615394 8564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:34.616370 8564 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:34.616636 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:34.617772 8564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:34.619557 8564 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:34.619634 8564 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:34.619776 8564 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:34.619837 8564 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:34.657847 8564 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43487
I20260812 19:55:34.657905 8718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43487 every 8 connection(s)
I20260812 19:55:34.659564 8564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:34.660152 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8564
I20260812 19:55:34.660601 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8564
I20260812 19:55:34.701213 8713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:34.792569 8719 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:34.792649 8722 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:34.792585 8720 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:34.793241 8722 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:34.793412 8719 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:34.794766 8719 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:34.794790 8722 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:34.793246 8720 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:34.795604 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8564
I20260812 19:55:34.818646 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7899
I20260812 19:55:34.827530 8055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:34.915978 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7899
I20260812 19:55:34.940305 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8075
I20260812 19:55:34.950246 8214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:35.042793 8226 heartbeater.cc:499] Master 127.25.14.61:46699 was elected leader, sending a full tablet report...
I20260812 19:55:35.045737 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8075
I20260812 19:55:35.071642 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8236
I20260812 19:55:35.078722 8314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:35.192337 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8236
I20260812 19:55:35.222048 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7501
I20260812 19:55:35.223318 7576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:35.335453 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7501
I20260812 19:55:35.369516 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7585
I20260812 19:55:35.370862 7657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:35.482811 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7585
Shutting down hive metastore.
2026-08-12T19:55:35Z chronyd exiting
2026-08-12T19:55:35Z chronyd exiting
2026-08-12T19:55:35Z chronyd exiting
2026-08-12T19:55:35Z chronyd exiting
2026-08-12T19:55:35Z chronyd exiting
2026-08-12T19:55:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:35Z Disabled control of system clock
2026-08-12T19:55:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:36Z Disabled control of system clock
2026-08-12T19:55:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:36Z Disabled control of system clock
2026-08-12T19:55:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:36Z Disabled control of system clock
2026-08-12T19:55:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:55:36Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:55:37: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:55:45.821870 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
W20260812 19:55:46.197786 8977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:46.198215 8977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:46.198385 8977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:46.207419 8977 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:46.207538 8977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:46.207597 8977 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:46.207706 8977 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:46.218811 8977 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:46.220636 8977 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:46.222791 8977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:46.232522 8983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:46.232923 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:46.233269 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:46.233438 8977 server_base.cc:1061] running on GCE node
I20260812 19:55:46.234330 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:46.236117 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:46.237385 8977 hybrid_clock.cc:648] HybridClock initialized: now 1786564546237223 us; error 123 us; skew 500 ppm
I20260812 19:55:46.237861 8977 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:46.240280 8977 webserver.cc:533] Webserver started at http://127.25.14.62:45149/ using document root <none> and password file <none>
I20260812 19:55:46.241103 8977 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:46.241261 8977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:46.247284 8977 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 19:55:46.250687 8994 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:46.252314 8977 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:55:46.252524 8977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:46.253334 8977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:46.269691 8977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:46.270754 8977 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:46.271104 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:46.294301 8977 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38251
I20260812 19:55:46.294320 9060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38251 every 8 connection(s)
I20260812 19:55:46.296223 8977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:46.303009 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8977
I20260812 19:55:46.303248 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:55:46.304260 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46699
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:46.304309 9061 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:46.308048 9061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap starting.
I20260812 19:55:46.329185 9061 log.cc:826] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:46.342917 9061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:46.343636 9061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap complete.
I20260812 19:55:46.354319 9061 raft_consensus.cc:359] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:46.355296 9061 raft_consensus.cc:740] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Initialized, Role: FOLLOWER
I20260812 19:55:46.356196 9061 consensus_queue.cc:260] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:46.357157 9065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:46.357548 9065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:46.358557 9061 sys_catalog.cc:565] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:46.369976 9079 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:46.370208 9079 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: loading cluster ID for follower catalog manager: success
I20260812 19:55:46.379702 9079 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: acquiring CA information for follower catalog manager: success
I20260812 19:55:46.384138 9079 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:46.396874 8977 master_runner.cc:187] Error getting master registration for 127.25.14.61:46699: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46699: connect: Connection refused (error 111),
I20260812 19:55:46.403045 8977 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:46.723819 9063 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:46.724282 9063 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:46.724429 9063 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:46.733433 9063 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:46.733600 9063 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:46.733718 9063 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:46.733825 9063 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:46.745177 9063 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:46.747205 9063 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:46.749531 9063 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:46.759393 9088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:46.759684 9089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:46.761063 9094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:46.762105 9063 server_base.cc:1061] running on GCE node
I20260812 19:55:46.763166 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:46.765142 9063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:46.768540 9063 hybrid_clock.cc:648] HybridClock initialized: now 1786564546768351 us; error 144 us; skew 500 ppm
I20260812 19:55:46.769035 9063 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:46.771955 9063 webserver.cc:533] Webserver started at http://127.25.14.61:37515/ using document root <none> and password file <none>
I20260812 19:55:46.772856 9063 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:46.773057 9063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:46.779723 9063 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:46.783632 9100 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:46.785336 9063 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:55:46.785597 9063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "a476e34d821e4ba7a38723a9919b5b68"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:46.786518 9063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:46.804934 9063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:46.806113 9063 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:46.806499 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:46.829874 9063 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46699
I20260812 19:55:46.829895 9156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46699 every 8 connection(s)
I20260812 19:55:46.831976 9063 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:55:46.839872 9157 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:46.842290 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9063
I20260812 19:55:46.842650 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:55:46.843634 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39357
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:46.843711 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Bootstrap starting.
I20260812 19:55:46.864472 9157 log.cc:826] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:46.879747 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:46.880511 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: Bootstrap complete.
I20260812 19:55:46.891624 9157 raft_consensus.cc:359] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:46.892791 9157 raft_consensus.cc:740] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a476e34d821e4ba7a38723a9919b5b68, State: Initialized, Role: FOLLOWER
I20260812 19:55:46.893826 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:46.894869 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:46.895212 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:46.896297 9157 sys_catalog.cc:565] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:46.908010 9175 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:46.908228 9175 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: loading cluster ID for follower catalog manager: success
I20260812 19:55:46.912707 9175 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: acquiring CA information for follower catalog manager: success
I20260812 19:55:46.915493 9175 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:46.947103 9063 master_runner.cc:187] Error getting master registration for 127.25.14.60:39357: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39357: connect: Connection refused (error 111),
I20260812 19:55:46.953197 9063 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:47.261431 9160 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:47.261917 9160 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:47.262063 9160 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:47.271150 9160 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:47.271312 9160 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:47.271421 9160 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:47.271530 9160 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:47.283453 9160 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:47.285311 9160 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:47.287511 9160 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:47.297086 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:47.298097 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:47.311306 9160 server_base.cc:1061] running on GCE node
W20260812 19:55:47.311298 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:47.312573 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:47.314627 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:47.315979 9160 hybrid_clock.cc:648] HybridClock initialized: now 1786564547315823 us; error 115 us; skew 500 ppm
I20260812 19:55:47.316414 9160 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:47.319029 9160 webserver.cc:533] Webserver started at http://127.25.14.60:37429/ using document root <none> and password file <none>
I20260812 19:55:47.319890 9160 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:47.320137 9160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:47.326216 9160 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:47.329478 9196 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:47.331095 9160 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:47.331314 9160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
format_stamp: "Formatted at 2026-08-12 19:55:29 on dist-test-slave-p3t5"
I20260812 19:55:47.332136 9160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:47.348565 9160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:47.349709 9160 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:47.350030 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:47.372022 9160 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:39357
I20260812 19:55:47.372030 9252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:39357 every 8 connection(s)
I20260812 19:55:47.374064 9160 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:55:47.380466 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9160
I20260812 19:55:47.380695 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:55:47.381855 9253 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:47.386539 9253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Bootstrap starting.
I20260812 19:55:47.407125 9253 log.cc:826] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:47.418390 9253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:47.419010 9253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Bootstrap complete.
I20260812 19:55:47.426017 9253 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:47.427098 9253 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdcf1ae0c8c45da8e8af22a4962ab4e, State: Initialized, Role: FOLLOWER
I20260812 19:55:47.427812 9253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:47.428755 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:47.429030 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:47.429944 9253 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:47.444262 9270 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:47.444510 9270 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: loading cluster ID for follower catalog manager: success
I20260812 19:55:47.450115 9270 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: acquiring CA information for follower catalog manager: success
I20260812 19:55:47.454388 9270 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:47.471415 9160 master_runner.cc:264] No leader master found from master 9fdcf1ae0c8c45da8e8af22a4962ab4e
I20260812 19:55:47.471573 9160 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:48.148721 9273 raft_consensus.cc:493] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:55:48.149067 9273 raft_consensus.cc:515] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:48.151161 9273 leader_election.cc:290] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:48.161705 9224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" is_pre_election: true
I20260812 19:55:48.161599 9129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68" is_pre_election: true
I20260812 19:55:48.162292 9224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 2.
I20260812 19:55:48.162286 9129 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 2.
I20260812 19:55:48.163097 9000 leader_election.cc:304] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e, a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters:
I20260812 19:55:48.163635 9273 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:55:48.163810 9273 raft_consensus.cc:493] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:55:48.163933 9273 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:55:48.168772 9273 raft_consensus.cc:515] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:48.169684 9273 leader_election.cc:290] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 3 election: Requested vote from peers a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:48.170176 9129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68"
I20260812 19:55:48.170195 9224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a020a9db64854326bffffac643bbfa5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e"
I20260812 19:55:48.170408 9129 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:55:48.170464 9224 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:55:48.174405 9129 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 3.
I20260812 19:55:48.174450 9224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a020a9db64854326bffffac643bbfa5d in term 3.
I20260812 19:55:48.175001 8998 leader_election.cc:304] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a020a9db64854326bffffac643bbfa5d, a476e34d821e4ba7a38723a9919b5b68; no voters:
I20260812 19:55:48.175390 9273 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:55:48.175848 9273 raft_consensus.cc:697] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 LEADER]: Becoming Leader. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Running, Role: LEADER
I20260812 19:55:48.176359 9273 consensus_queue.cc:237] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:48.178936 9280 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.179288 9280 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:48.179991 9282 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:48.182199 9282 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:48.183043 9282 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:48.183122 9282 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:48.184234 9282 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:48.185026 9282 catalog_manager.cc:6103] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Loaded TSK: 0
I20260812 19:55:48.186070 9282 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:55:48.211129 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:55:48.606839 9283 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:48.607275 9283 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:48.607432 9283 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:48.616320 9283 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:48.616604 9283 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:48.627380 9283 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:48.629242 9283 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:48.631493 9283 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:48.643090 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:48.644089 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:48.644923 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:48.646065 9283 server_base.cc:1061] running on GCE node
I20260812 19:55:48.646785 9283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:48.648133 9283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:48.649515 9283 hybrid_clock.cc:648] HybridClock initialized: now 1786564548649361 us; error 102 us; skew 500 ppm
I20260812 19:55:48.649976 9283 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:48.652746 9283 webserver.cc:533] Webserver started at http://127.25.14.1:40857/ using document root <none> and password file <none>
I20260812 19:55:48.653688 9283 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:48.653817 9283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:48.660142 9283 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 19:55:48.663686 9300 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:48.665232 9283 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:55:48.665448 9283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:48.666296 9283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:48.687794 9283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:48.688803 9283 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:48.689168 9283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:48.690347 9283 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:48.692272 9283 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:48.692351 9283 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:48.692598 9283 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:48.692662 9283 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 19:55:48.704443 9129 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 3 FOLLOWER]: Refusing update from remote peer a020a9db64854326bffffac643bbfa5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:55:48.705780 9329 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:55:48.710834 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.711153 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:48.726224 9224 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Refusing update from remote peer a020a9db64854326bffffac643bbfa5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:55:48.729849 9329 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:55:48.734472 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.735579 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:48.741276 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.741601 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:48.744990 9283 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36243
I20260812 19:55:48.745150 9438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36243 every 8 connection(s)
I20260812 19:55:48.746925 9283 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:48.753998 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9283
I20260812 19:55:48.754273 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:55:48.762552 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:48.787513 9439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a020a9db64854326bffffac643bbfa5d. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.787912 9439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:48.789755 9454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:48.790108 9454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:48.800463 9441 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:48.800642 9443 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:48.801028 9441 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:48.801033 9443 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:48.802296 9443 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:48.802410 9441 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:48.803293 9440 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:48.803576 9440 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:48.804234 9440 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:48.805480 9117 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:48.806398 9019 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:48.809100 9019 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:51127
I20260812 19:55:48.810883 9214 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
W20260812 19:55:49.180557 9455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:49.181022 9455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:49.181174 9455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:49.190572 9455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:49.190843 9455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:55:49.201733 9455 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:49.203603 9455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:49.205821 9455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:49.217356 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:49.217399 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:49.219298 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:49.219688 9455 server_base.cc:1061] running on GCE node
I20260812 19:55:49.220413 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:49.221815 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:49.223117 9455 hybrid_clock.cc:648] HybridClock initialized: now 1786564549222964 us; error 117 us; skew 500 ppm
I20260812 19:55:49.223503 9455 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:49.226025 9455 webserver.cc:533] Webserver started at http://127.25.14.2:34195/ using document root <none> and password file <none>
I20260812 19:55:49.226919 9455 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:49.227069 9455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:49.233593 9455 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:55:49.236822 9471 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.238653 9455 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 19:55:49.238893 9455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1cb884363ca343429823b38dbcfc1e1f"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:49.239717 9455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:49.253185 9329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a020a9db64854326bffffac643bbfa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:49.253494 9329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:49.256397 9455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:49.257357 9455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:49.257752 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:49.258870 9455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:49.260903 9455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:49.260982 9455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.261153 9455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:49.261214 9455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.305361 9455 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:35565
I20260812 19:55:49.305421 9607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:35565 every 8 connection(s)
I20260812 19:55:49.307190 9455 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:55:49.317803 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9455
I20260812 19:55:49.318018 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 19:55:49.324509 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:49.327975 9610 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:49.328111 9608 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:49.328137 9609 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:49.328562 9610 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.328557 9609 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.328557 9608 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.330251 9608 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:49.330492 9610 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:49.329735 9609 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:49.332283 9214 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:35565)
I20260812 19:55:49.332307 9117 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:35565)
I20260812 19:55:49.332598 9019 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:35565)
I20260812 19:55:49.333863 9019 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:41469
W20260812 19:55:49.727672 9619 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:49.728116 9619 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:49.728271 9619 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:49.737417 9619 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:49.737701 9619 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:55:49.751699 9619 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:49.754074 9619 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:49.756548 9619 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:49.768432 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:49.768548 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:49.768524 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:49.769769 9619 server_base.cc:1061] running on GCE node
I20260812 19:55:49.770562 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:49.772071 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:49.775470 9619 hybrid_clock.cc:648] HybridClock initialized: now 1786564549775247 us; error 159 us; skew 500 ppm
I20260812 19:55:49.775959 9619 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:49.778981 9619 webserver.cc:533] Webserver started at http://127.25.14.3:36177/ using document root <none> and password file <none>
I20260812 19:55:49.780089 9619 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:49.780237 9619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:49.787258 9619 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:49.790844 9637 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.792531 9619 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:55:49.792752 9619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "51c635028a6f41de8592e229bcdf3fa6"
format_stamp: "Formatted at 2026-08-12 19:55:31 on dist-test-slave-p3t5"
I20260812 19:55:49.793743 9619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:49.812781 9441 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:49.828941 9619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:49.830037 9619 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:49.830410 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:49.831661 9619 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:49.833811 9619 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:49.833920 9619 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.834090 9619 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:49.834172 9619 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:49.873307 9619 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:38621
I20260812 19:55:49.873353 9774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:38621 every 8 connection(s)
I20260812 19:55:49.875156 9619 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:55:49.882361 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9619
I20260812 19:55:49.882653 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 19:55:49.907697 9776 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:49.907964 9775 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:49.908214 9777 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:49.908237 9776 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.908420 9777 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.908241 9775 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:49.909747 9775 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:49.909791 9777 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:49.909834 9776 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:49.911772 9019 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38621)
I20260812 19:55:49.912019 9213 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38621)
I20260812 19:55:49.912036 9117 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38621)
I20260812 19:55:49.912882 9019 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:48101
I20260812 19:55:49.928402 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:49.937071 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:55:49.937304 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8977
I20260812 19:55:49.938406 9054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:50.056095 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8977
I20260812 19:55:50.086117 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--webserver_interface=127.25.14.62
--webserver_port=45149
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357 with env {}
I20260812 19:55:50.336830 9609 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
W20260812 19:55:50.338910 9609 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:38251 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:38251: connect: Connection refused (error 111)
W20260812 19:55:50.477481 9787 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:50.477917 9787 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:50.478060 9787 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:50.487087 9787 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:50.487191 9787 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:50.487247 9787 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:50.487377 9787 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:50.498279 9787 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=45149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:50.500326 9787 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:50.502442 9787 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:50.511984 9794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:50.512022 9795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:50.512737 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:50.514066 9787 server_base.cc:1061] running on GCE node
I20260812 19:55:50.514925 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:50.516695 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:50.518047 9787 hybrid_clock.cc:648] HybridClock initialized: now 1786564550517890 us; error 120 us; skew 500 ppm
I20260812 19:55:50.518445 9787 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:50.521085 9787 webserver.cc:533] Webserver started at http://127.25.14.62:45149/ using document root <none> and password file <none>
I20260812 19:55:50.522055 9787 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:50.522228 9787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:50.528319 9787 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:55:50.532094 9805 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:50.533968 9787 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:50.534197 9787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "a020a9db64854326bffffac643bbfa5d"
format_stamp: "Formatted at 2026-08-12 19:55:28 on dist-test-slave-p3t5"
I20260812 19:55:50.535033 9787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:50.565044 9787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:50.566427 9787 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:50.566814 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:50.588806 9787 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38251
I20260812 19:55:50.588825 9868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38251 every 8 connection(s)
I20260812 19:55:50.590760 9787 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:50.598270 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9787
I20260812 19:55:50.598595 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9283
I20260812 19:55:50.598964 9870 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:55:50.604024 9870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap starting.
I20260812 19:55:50.609195 9434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:50.625342 9870 log.cc:826] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:50.636494 9870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:55:50.637189 9870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: Bootstrap complete.
I20260812 19:55:50.645931 9870 raft_consensus.cc:359] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:50.646847 9870 raft_consensus.cc:740] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: a020a9db64854326bffffac643bbfa5d, State: Initialized, Role: FOLLOWER
I20260812 19:55:50.647848 9870 consensus_queue.cc:260] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:50.648910 9877 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:50.649245 9877 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:50.650337 9870 sys_catalog.cc:565] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:50.662683 9891 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:50.662899 9891 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: loading cluster ID for follower catalog manager: success
I20260812 19:55:50.668670 9891 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: acquiring CA information for follower catalog manager: success
I20260812 19:55:50.672861 9891 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:50.708328 9787 master_runner.cc:264] No leader master found from master a020a9db64854326bffffac643bbfa5d
I20260812 19:55:50.708547 9787 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:50.722036 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9283
I20260812 19:55:50.754602 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36243
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=40857
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:50.915369 9776 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:50.927129 9824 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "51c635028a6f41de8592e229bcdf3fa6" instance_seqno: 1786564549864422) as {username='slave'} at 127.25.14.3:46811; Asking this server to re-register.
I20260812 19:55:50.928362 9776 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:50.930446 9824 ts_manager.cc:194] Registered new tserver with Master: 51c635028a6f41de8592e229bcdf3fa6 (127.25.14.3:38621)
I20260812 19:55:51.139524 9899 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:51.139835 9899 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:51.142686 9899 leader_election.cc:290] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699)
I20260812 19:55:51.143828 9129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68" is_pre_election: true
I20260812 19:55:51.153966 9836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d" is_pre_election: true
I20260812 19:55:51.154650 9836 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in term 3.
I20260812 19:55:51.155447 9198 leader_election.cc:304] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e, a020a9db64854326bffffac643bbfa5d; no voters: a476e34d821e4ba7a38723a9919b5b68
I20260812 19:55:51.155944 9899 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:55:51.156100 9899 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Starting leader election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:51.156272 9899 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:55:51.158874 9899 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:51.159696 9899 leader_election.cc:290] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 4 election: Requested vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), a476e34d821e4ba7a38723a9919b5b68 (127.25.14.61:46699)
I20260812 19:55:51.160329 9836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d"
I20260812 19:55:51.160435 9129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a476e34d821e4ba7a38723a9919b5b68"
I20260812 19:55:51.160573 9836 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:55:51.164664 9836 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in term 4.
I20260812 19:55:51.165328 9198 leader_election.cc:304] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e, a020a9db64854326bffffac643bbfa5d; no voters: a476e34d821e4ba7a38723a9919b5b68
I20260812 19:55:51.165727 9899 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:55:51.166231 9899 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [term 4 LEADER]: Becoming Leader. State: Replica: 9fdcf1ae0c8c45da8e8af22a4962ab4e, State: Running, Role: LEADER
I20260812 19:55:51.166716 9899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:51.168695 9901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdcf1ae0c8c45da8e8af22a4962ab4e. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.169111 9901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:51.169963 9903 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:51.172034 9903 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:55:51.172909 9903 catalog_manager.cc:1295] Loaded cluster ID: 2bed31ed0d7e4731bd7642618ab7a021
I20260812 19:55:51.173024 9903 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:51.174300 9903 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:51.175127 9903 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e: Loaded TSK: 0
I20260812 19:55:51.176061 9903 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260812 19:55:51.190294 9897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:51.190667 9897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:51.190804 9897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:51.199834 9897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:51.200095 9897 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:51.210798 9897 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40857
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:51.212510 9897 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:51.214457 9897 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:51.224926 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:51.225049 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:51.225844 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:51.226378 9897 server_base.cc:1061] running on GCE node
I20260812 19:55:51.227069 9897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:51.228309 9897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:51.229565 9897 hybrid_clock.cc:648] HybridClock initialized: now 1786564551229421 us; error 111 us; skew 500 ppm
I20260812 19:55:51.229956 9897 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:51.232474 9897 webserver.cc:533] Webserver started at http://127.25.14.1:40857/ using document root <none> and password file <none>
I20260812 19:55:51.233378 9897 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:51.233551 9897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:51.240187 9897 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 19:55:51.243865 9918 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:51.245743 9897 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:51.245956 9897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:51.246793 9897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:51.269532 9897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:51.270673 9897 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:51.271035 9897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:51.272245 9897 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:51.274433 9897 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:51.274518 9897 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:51.274660 9897 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:51.274727 9897 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:51.314698 9897 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36243
I20260812 19:55:51.314724 10050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36243 every 8 connection(s)
I20260812 19:55:51.316504 9897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:51.323102 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9897
I20260812 19:55:51.356957 10052 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:51.357460 10052 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:51.358980 10052 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:51.361665 9824 ts_manager.cc:194] Registered new tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:51.365712 10051 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:51.365883 10051 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:51.366510 10051 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:51.367606 10053 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46699
I20260812 19:55:51.367667 9609 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:51.367754 10053 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:51.367878 9609 heartbeater.cc:499] Master 127.25.14.62:38251 was elected leader, sending a full tablet report...
I20260812 19:55:51.367875 9213 ts_manager.cc:194] Re-registered known tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:51.368245 10053 heartbeater.cc:507] Master 127.25.14.61:46699 requested a full tablet report, sending...
I20260812 19:55:51.369184 9824 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1cb884363ca343429823b38dbcfc1e1f" instance_seqno: 1786564549292423) as {username='slave'} at 127.25.14.2:55057; Asking this server to re-register.
I20260812 19:55:51.369762 9117 ts_manager.cc:194] Re-registered known tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:51.369865 9213 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:54265
I20260812 19:55:51.370499 9609 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:51.371794 9824 ts_manager.cc:194] Registered new tserver with Master: 1cb884363ca343429823b38dbcfc1e1f (127.25.14.2:35565)
I20260812 19:55:51.406109 10064 raft_consensus.cc:493] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a020a9db64854326bffffac643bbfa5d)
I20260812 19:55:51.406387 10064 raft_consensus.cc:515] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } }
I20260812 19:55:51.408987 10064 leader_election.cc:290] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a020a9db64854326bffffac643bbfa5d (127.25.14.62:38251), 9fdcf1ae0c8c45da8e8af22a4962ab4e (127.25.14.60:39357)
I20260812 19:55:51.433490 9836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a020a9db64854326bffffac643bbfa5d" is_pre_election: true
I20260812 19:55:51.434006 9836 raft_consensus.cc:2393] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a476e34d821e4ba7a38723a9919b5b68 in current term 4: Already voted for candidate 9fdcf1ae0c8c45da8e8af22a4962ab4e in this term.
I20260812 19:55:51.435276 9224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a476e34d821e4ba7a38723a9919b5b68" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" is_pre_election: true
I20260812 19:55:51.436120 9104 leader_election.cc:304] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a476e34d821e4ba7a38723a9919b5b68; no voters: 9fdcf1ae0c8c45da8e8af22a4962ab4e, a020a9db64854326bffffac643bbfa5d
I20260812 19:55:51.437006 10064 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:55:51.440460 10064 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:55:51.511427 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9897
I20260812 19:55:51.543260 10045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:51.570721 9129 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [term 4 FOLLOWER]: Refusing update from remote peer 9fdcf1ae0c8c45da8e8af22a4962ab4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260812 19:55:51.571609 9901 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 19:55:51.573781 10064 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdcf1ae0c8c45da8e8af22a4962ab4e. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.574093 10064 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:51.576103 9899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.576370 9899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:51.577468 9836 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [term 4 FOLLOWER]: Refusing update from remote peer 9fdcf1ae0c8c45da8e8af22a4962ab4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260812 19:55:51.577518 10064 sys_catalog.cc:455] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.577762 10064 sys_catalog.cc:458] T 00000000000000000000000000000000 P a476e34d821e4ba7a38723a9919b5b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:51.578384 9901 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 19:55:51.582497 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.582973 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:51.583541 10076 sys_catalog.cc:455] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdcf1ae0c8c45da8e8af22a4962ab4e. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.583825 10076 sys_catalog.cc:458] T 00000000000000000000000000000000 P a020a9db64854326bffffac643bbfa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:51.644490 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9897
I20260812 19:55:51.671855 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36243
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=40857
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260812 19:55:51.962365 9901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a020a9db64854326bffffac643bbfa5d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38251 } } peers { permanent_uuid: "a476e34d821e4ba7a38723a9919b5b68" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46699 } } peers { permanent_uuid: "9fdcf1ae0c8c45da8e8af22a4962ab4e" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39357 } } }
I20260812 19:55:51.962677 9901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdcf1ae0c8c45da8e8af22a4962ab4e [sys.catalog]: This master's current role is: LEADER
W20260812 19:55:52.068912 10080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:52.069273 10080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:52.069418 10080 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:55:52.069523 10080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:52.078529 10080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:52.078783 10080 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:52.089498 10080 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40857
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:52.091404 10080 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:52.093533 10080 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:52.104831 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:52.105187 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:52.105379 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:52.106130 10080 server_base.cc:1061] running on GCE node
I20260812 19:55:52.106779 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:52.108026 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:52.111377 10080 hybrid_clock.cc:648] HybridClock initialized: now 1786564552111199 us; error 116 us; skew 500 ppm
I20260812 19:55:52.111781 10080 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:52.114513 10080 webserver.cc:533] Webserver started at http://127.25.14.1:40857/ using document root <none> and password file <none>
I20260812 19:55:52.115403 10080 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:52.115582 10080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:52.121803 10080 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:52.125172 10097 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:52.126981 10080 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:52.127218 10080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:52.128080 10080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:52.153414 10080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:52.154498 10080 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:52.154827 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:52.155961 10080 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:52.157886 10080 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:52.158025 10080 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:52.158170 10080 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:52.158277 10080 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:55:52.158691 10080 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:55:52.158818 10080 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:55:52.158911 10080 fault_injection.cc:59] injecting fault for kudu (pid 10080): FLAGS_fault_before_start (process will exit)
I20260812 19:55:52.183851 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:36243
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--webserver_port=40857
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 19:55:52.339847 9608 heartbeater.cc:499] Master 127.25.14.60:39357 was elected leader, sending a full tablet report...
W20260812 19:55:52.583809 10109 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:52.584153 10109 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:52.584345 10109 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:52.593375 10109 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:52.593611 10109 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:52.604379 10109 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36231,127.25.14.21:43507,127.25.14.22:36053,127.25.14.23:44853,127.25.14.24:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:36243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40857
--tserver_master_addrs=127.25.14.62:38251,127.25.14.61:46699,127.25.14.60:39357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:52.606256 10109 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:52.608301 10109 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:52.619366 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:52.619366 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:52.619913 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:52.620482 10109 server_base.cc:1061] running on GCE node
I20260812 19:55:52.621197 10109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:55:52.622546 10109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:55:52.624009 10109 hybrid_clock.cc:648] HybridClock initialized: now 1786564552623820 us; error 132 us; skew 500 ppm
I20260812 19:55:52.624406 10109 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:55:52.626991 10109 webserver.cc:533] Webserver started at http://127.25.14.1:40857/ using document root <none> and password file <none>
I20260812 19:55:52.627909 10109 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:52.628088 10109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:52.635470 10109 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 19:55:52.638998 10123 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:52.640595 10109 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:52.640832 10109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "64f0a83d78fb404fa215f6e6efdc574d"
format_stamp: "Formatted at 2026-08-12 19:55:30 on dist-test-slave-p3t5"
I20260812 19:55:52.641820 10109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:52.680598 10109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:52.681697 10109 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:52.682072 10109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:52.683189 10109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:52.685320 10109 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:52.685444 10109 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:52.685611 10109 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:52.685732 10109 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 19:55:52.723970 10109 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36243
I20260812 19:55:52.724006 10260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36243 every 8 connection(s)
I20260812 19:55:52.725899 10109 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:52.736563 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10109
I20260812 19:55:52.736984 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10109
I20260812 19:55:52.759033 10261 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39357
I20260812 19:55:52.759496 10261 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:52.759945 10262 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38251
I20260812 19:55:52.760149 10262 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:52.760386 10256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:52.850016 10261 heartbeater.cc:507] Master 127.25.14.60:39357 requested a full tablet report, sending...
I20260812 19:55:52.851637 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10109
I20260812 19:55:52.851648 10262 heartbeater.cc:507] Master 127.25.14.62:38251 requested a full tablet report, sending...
I20260812 19:55:52.852641 9213 ts_manager.cc:194] Re-registered known tserver with Master: 64f0a83d78fb404fa215f6e6efdc574d (127.25.14.1:36243)
I20260812 19:55:52.853838 9213 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:44735
W20260812 19:55:52.876978 9197 connection.cc:573] server connection from 127.25.14.1:44735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:55:52.877601 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9455
I20260812 19:55:52.886452 9600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:52.918483 9775 heartbeater.cc:499] Master 127.25.14.60:39357 was elected leader, sending a full tablet report...
I20260812 19:55:52.977033 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9455
I20260812 19:55:53.001457 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9619
I20260812 19:55:53.010129 9768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:53.101409 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9619
I20260812 19:55:53.126797 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9787
I20260812 19:55:53.133373 9864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:53.242535 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9787
I20260812 19:55:53.271579 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9063
I20260812 19:55:53.272887 9152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:53.385984 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9063
I20260812 19:55:53.414378 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9160
I20260812 19:55:53.415650 9248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:55:53.528558 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9160
Shutting down hive metastore.
2026-08-12T19:55:53Z chronyd exiting
2026-08-12T19:55:53Z chronyd exiting
2026-08-12T19:55:53Z chronyd exiting
2026-08-12T19:55:53Z chronyd exiting
2026-08-12T19:55:53Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/5 (42728 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/6
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:55:54 dist-test-slave-p3t5 krb5kdc[10292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:55:54 dist-test-slave-p3t5 krb5kdc[10292](info): set up 2 sockets
Aug 12 19:55:54 dist-test-slave-p3t5 krb5kdc[10292](info): commencing operation
krb5kdc: starting...
W20260812 19:55:56.184754 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.165s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:55:56 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:56.509987 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:55:56.894230 10309 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:56.894699 10309 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:56.894881 10309 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:56.903687 10309 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:56.903893 10309 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:56.904036 10309 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:56.904124 10309 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:56.915449 10309 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:56.917413 10309 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:56.919036 10309 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:56.928417 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:56.928475 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:56.929598 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:56.929961 10309 server_base.cc:1061] running on GCE node
I20260812 19:55:56.930909 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:55:56.931123 10309 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:55:56.931232 10309 hybrid_clock.cc:648] HybridClock initialized: now 1786564556931208 us; error 0 us; skew 500 ppm
I20260812 19:55:56.931620 10309 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:55:56 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:55:56.935513 10309 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:55:56.937894 10309 webserver.cc:533] Webserver started at http://127.25.14.62:34339/ using document root <none> and password file <none>
I20260812 19:55:56.938750 10309 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:56.938898 10309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:56.939301 10309 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:56.942080 10309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:55:56.943010 10309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:55:56.949196 10309 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:55:56.953306 10326 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:56.954955 10309 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 19:55:56.955190 10309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:55:56.955423 10309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:56.979080 10309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:56.983527 10309 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:56.983850 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:57.006743 10309 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33389
I20260812 19:55:57.006757 10391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33389 every 8 connection(s)
I20260812 19:55:57.008567 10309 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:55:57.013795 10392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:55:57.014110 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10309
I20260812 19:55:57.014338 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:55:57.023808 10392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:57 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 19:55:57.052613 10392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:57.054148 10330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
W20260812 19:55:57.056104 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:57.105537 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } attempt: 1
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
W20260812 19:55:57.107957 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:57.145381 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41507
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:55:57.176301 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } attempt: 2
W20260812 19:55:57.178588 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
I20260812 19:55:57.269938 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } attempt: 3
W20260812 19:55:57.272270 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
I20260812 19:55:57.415645 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } attempt: 4
W20260812 19:55:57.418128 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111)
W20260812 19:55:57.526628 10403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:57.527082 10403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:57.527244 10403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:57.536296 10403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:57.536425 10403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:57.536490 10403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:57.536535 10403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:57.548142 10403 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:57.550031 10403 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:57.551508 10403 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:57.561085 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:57.560806 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:57.561362 10412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:57.562081 10403 server_base.cc:1061] running on GCE node
I20260812 19:55:57.562984 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:55:57.563192 10403 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:55:57.563306 10403 hybrid_clock.cc:648] HybridClock initialized: now 1786564557563281 us; error 0 us; skew 500 ppm
I20260812 19:55:57.563690 10403 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:55:57 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564557, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:55:57.567445 10403 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:55:57.569697 10403 webserver.cc:533] Webserver started at http://127.25.14.61:36855/ using document root <none> and password file <none>
I20260812 19:55:57.570514 10403 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:57.570663 10403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:57.571055 10403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:57.573611 10403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "4d385919465b42b3b0ad831bcdb77f91"
format_stamp: "Formatted at 2026-08-12 19:55:57 on dist-test-slave-p3t5"
I20260812 19:55:57.574661 10403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "4d385919465b42b3b0ad831bcdb77f91"
format_stamp: "Formatted at 2026-08-12 19:55:57 on dist-test-slave-p3t5"
I20260812 19:55:57.580438 10403 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:55:57.584368 10419 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:57.586472 10403 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:55:57.586758 10403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "4d385919465b42b3b0ad831bcdb77f91"
format_stamp: "Formatted at 2026-08-12 19:55:57 on dist-test-slave-p3t5"
I20260812 19:55:57.587069 10403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:57.611889 10403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:57.616364 10403 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:57.616691 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:57.639133 10403 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:41507
I20260812 19:55:57.639158 10478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:41507 every 8 connection(s)
I20260812 19:55:57.640925 10403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:55:57.646000 10479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:55:57.647441 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10403
I20260812 19:55:57.647701 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:55:57.655983 10479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:57 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564557, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
I20260812 19:55:57.680938 10479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:57 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564557, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:55:57.700234 10479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:57.701563 10421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
W20260812 19:55:57.703570 10479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:57.717456 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } attempt: 5
Aug 12 19:55:57 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:55:57.733697 10392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:55:57.735946 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 19:55:57.752944 10479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 1
W20260812 19:55:57.755750 10479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:57.779605 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45069
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:55:57.787236 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 1
W20260812 19:55:57.789763 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:57.824010 10479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 2
W20260812 19:55:57.826292 10479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:57.858093 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 2
W20260812 19:55:57.860456 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:57.917536 10479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 3
W20260812 19:55:57.919955 10479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:57.966777 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 3
W20260812 19:55:57.969292 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:58.063247 10479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 4
W20260812 19:55:58.065838 10479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
I20260812 19:55:58.146564 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 4
W20260812 19:55:58.149134 10392 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111)
W20260812 19:55:58.159658 10493 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:58.160140 10493 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:58.160307 10493 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:58.171608 10493 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:55:58.171772 10493 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:58.171867 10493 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:55:58.171957 10493 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:55:58.184487 10493 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:58.186331 10493 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:58.187764 10493 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:58.196636 10499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:58.196805 10500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:58.197463 10502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:58.198077 10493 server_base.cc:1061] running on GCE node
I20260812 19:55:58.199020 10493 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:55:58.199245 10493 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:55:58.199358 10493 hybrid_clock.cc:648] HybridClock initialized: now 1786564558199334 us; error 0 us; skew 500 ppm
I20260812 19:55:58.199759 10493 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564558, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:55:58.203608 10493 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:55:58.206067 10493 webserver.cc:533] Webserver started at http://127.25.14.60:36699/ using document root <none> and password file <none>
I20260812 19:55:58.206971 10493 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:58.207135 10493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:58.207569 10493 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:58.210592 10493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
format_stamp: "Formatted at 2026-08-12 19:55:58 on dist-test-slave-p3t5"
I20260812 19:55:58.211544 10493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
format_stamp: "Formatted at 2026-08-12 19:55:58 on dist-test-slave-p3t5"
I20260812 19:55:58.217975 10493 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:55:58.222250 10510 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:58.224045 10493 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:55:58.224344 10493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
format_stamp: "Formatted at 2026-08-12 19:55:58 on dist-test-slave-p3t5"
I20260812 19:55:58.224593 10493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:58.241144 10493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:58.251807 10493 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:58.252179 10493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:58.276262 10493 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:45069
I20260812 19:55:58.276293 10573 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:45069 every 8 connection(s)
I20260812 19:55:58.278131 10493 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:55:58.280807 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10493
I20260812 19:55:58.281060 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:55:58.283854 10574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:55:58.296664 10574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564558, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:55:58.324101 10574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564558, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:55:58.339298 10574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564558, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:55:58.358482 10574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Bootstrap starting.
I20260812 19:55:58.362665 10574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:58.363838 10574 log.cc:826] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:58.365120 10479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 5
I20260812 19:55:58.367091 10574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: No bootstrap required, opened a new log
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564557, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:55:58.374114 10574 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.374529 10574 raft_consensus.cc:385] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:58.374687 10574 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a4dfd879ee4b95b2f75d60c4249a92, State: Initialized, Role: FOLLOWER
I20260812 19:55:58.375491 10574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.377879 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.378219 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.379447 10574 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:55:58.388643 10479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Bootstrap starting.
I20260812 19:55:58.393525 10479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:58.395057 10479 log.cc:826] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:58.389173 10493 master_runner.cc:187] Error getting master registration for 127.25.14.62:33389: OK, instance_id { permanent_uuid: "721a195de0494d6a83841e456c57556a" instance_seqno: 1786564557000456 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:55:58.396099 10493 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:55:58.398964 10479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: No bootstrap required, opened a new log
W20260812 19:55:58.399397 10600 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:58.399544 10600 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:55:58.407220 10479 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.407799 10479 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:58.408007 10479 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d385919465b42b3b0ad831bcdb77f91, State: Initialized, Role: FOLLOWER
I20260812 19:55:58.408994 10479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.410225 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.410535 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.411754 10479 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:55:58.423357 10614 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:58.423599 10614 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:55:58.426433 10392 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } attempt: 5
I20260812 19:55:58.422458 10403 master_runner.cc:187] Error getting master registration for 127.25.14.62:33389: OK, instance_id { permanent_uuid: "721a195de0494d6a83841e456c57556a" instance_seqno: 1786564557000456 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:55:58.428018 10403 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 19:55:58 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:55:58.447162 10392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap starting.
I20260812 19:55:58.450919 10392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Neither blocks nor log segments found. Creating new log.
I20260812 19:55:58.452541 10392 log.cc:826] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Log is configured to *not* fsync() on all Append() calls
I20260812 19:55:58.459230 10392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: No bootstrap required, opened a new log
I20260812 19:55:58.465024 10392 raft_consensus.cc:359] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.465370 10392 raft_consensus.cc:385] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:55:58.465538 10392 raft_consensus.cc:740] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721a195de0494d6a83841e456c57556a, State: Initialized, Role: FOLLOWER
I20260812 19:55:58.466380 10392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.467590 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.467988 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.468912 10392 sys_catalog.cc:565] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:55:58.487571 10333 tablet.cc:2406] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:55:58.496270 10309 master_runner.cc:264] No leader master found from master 721a195de0494d6a83841e456c57556a
I20260812 19:55:58.496448 10309 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:55:58.499454 10630 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:55:58.499738 10630 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:55:58.504684 10517 tablet.cc:2406] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:55:58.539289 10603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:55:58.539654 10603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.541889 10603 leader_election.cc:290] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721a195de0494d6a83841e456c57556a (127.25.14.62:33389), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:55:58.543596 10544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" is_pre_election: true
I20260812 19:55:58.544224 10544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 0.
I20260812 19:55:58.545655 10359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721a195de0494d6a83841e456c57556a" is_pre_election: true
I20260812 19:55:58.546293 10359 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 0.
I20260812 19:55:58.547227 10421 leader_election.cc:304] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters:
I20260812 19:55:58.548316 10603 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:55:58.548484 10603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:55:58.548590 10603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:58.556712 10603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.560562 10359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721a195de0494d6a83841e456c57556a"
I20260812 19:55:58.560828 10359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:58.571822 10359 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 1.
I20260812 19:55:58.572737 10421 leader_election.cc:304] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters:
I20260812 19:55:58.574611 10603 leader_election.cc:290] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 1 election: Requested vote from peers 721a195de0494d6a83841e456c57556a (127.25.14.62:33389), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:55:58.575356 10603 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:55:58.576009 10603 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 LEADER]: Becoming Leader. State: Replica: 4d385919465b42b3b0ad831bcdb77f91, State: Running, Role: LEADER
I20260812 19:55:58.576920 10603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:55:58.598076 10544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
I20260812 19:55:58.598347 10544 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:55:58.602075 10544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 1.
I20260812 19:55:58.594439 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.604691 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:58.611784 10634 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:55:58.614707 10634 catalog_manager.cc:1520] Initializing Kudu cluster ID...
W20260812 19:55:58.619776 10426 tablet.cc:2406] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:55:58.636256 10544 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Refusing update from remote peer 4d385919465b42b3b0ad831bcdb77f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:55:58.637565 10603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:55:58.660303 10359 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Refusing update from remote peer 4d385919465b42b3b0ad831bcdb77f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:55:58.661649 10603 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:55:58.698987 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.699316 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.714285 10635 mvcc.cc:204] Tried to move back new op lower bound from 7317768432160047104 to 7317768431963275264. Current Snapshot: MvccSnapshot[applied={T|T < 7317768432160047104}]
I20260812 19:55:58.726366 10637 mvcc.cc:204] Tried to move back new op lower bound from 7317768432160047104 to 7317768431963275264. Current Snapshot: MvccSnapshot[applied={T|T < 7317768432160047104}]
I20260812 19:55:58.738641 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.738941 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.746797 10636 mvcc.cc:204] Tried to move back new op lower bound from 7317768432160047104 to 7317768431963275264. Current Snapshot: MvccSnapshot[applied={T|T < 7317768432160047104}]
I20260812 19:55:58.747915 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.748175 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.750597 10634 catalog_manager.cc:1383] Generated new cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:55:58.750717 10634 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:55:58.759326 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.759603 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:55:58.771050 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.771364 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:58.771807 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:55:58.772019 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
I20260812 19:55:58.798838 10634 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:55:58.801023 10634 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:55:58.856164 10634 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Generated new TSK 0
I20260812 19:55:58.857718 10634 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:59.044000 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:55:59.401727 10600 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:55:59.401831 10600 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: loading cluster ID for follower catalog manager: success
I20260812 19:55:59.404597 10600 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: acquiring CA information for follower catalog manager: success
I20260812 19:55:59.406711 10600 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:55:59.437740 10647 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:55:59.438161 10647 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:55:59.438333 10647 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:55:59.446878 10647 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:55:59.447166 10647 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:55:59.458112 10647 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:55:59.459992 10647 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:55:59.461658 10647 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:55:59.472836 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:59.472774 10653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:55:59.473688 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:55:59.474308 10647 server_base.cc:1061] running on GCE node
I20260812 19:55:59.475023 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:55:59.475230 10647 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:55:59.475313 10647 hybrid_clock.cc:648] HybridClock initialized: now 1786564559475294 us; error 0 us; skew 500 ppm
I20260812 19:55:59.475641 10647 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:55:59 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564559, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:55:59.479514 10647 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:55:59.481491 10647 webserver.cc:533] Webserver started at http://127.25.14.1:34627/ using document root <none> and password file <none>
I20260812 19:55:59.482372 10647 fs_manager.cc:362] Metadata directory not provided
I20260812 19:55:59.482479 10647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:55:59.482941 10647 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:55:59.485924 10647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:55:59.486847 10647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:55:59.493264 10647 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 19:55:59.497087 10663 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:59.498894 10647 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:55:59.499095 10647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:55:59.499325 10647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:55:59.501767 10630 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:55:59.501875 10630 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: loading cluster ID for follower catalog manager: success
I20260812 19:55:59.504520 10630 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: acquiring CA information for follower catalog manager: success
I20260812 19:55:59.506769 10630 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:55:59.516930 10647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:55:59.522190 10647 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:55:59.522583 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:55:59.523809 10647 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:55:59.525786 10647 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:55:59.525887 10647 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:59.526082 10647 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:55:59.526177 10647 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:55:59.566448 10647 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39527
I20260812 19:55:59.566504 10804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39527 every 8 connection(s)
I20260812 19:55:59.568132 10647 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:55:59.571426 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10647
I20260812 19:55:59.571725 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:55:59 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564559, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:55:59 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564559, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:55:59 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564559, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:55:59.602954 10807 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:55:59.603477 10807 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:59.604601 10807 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:55:59.605767 10806 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:55:59.605993 10806 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:59.606511 10806 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:55:59.606935 10805 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:55:59.607079 10805 heartbeater.cc:461] Registering TS with master...
I20260812 19:55:59.607538 10439 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
I20260812 19:55:59.607601 10805 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:55:59.608605 10345 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
I20260812 19:55:59.609995 10439 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:58779
I20260812 19:55:59.611009 10532 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:55:59.713421 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:00.099711 10824 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:00.100157 10824 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:00.100322 10824 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:00.109535 10824 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:00.109833 10824 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:00.121308 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:00.123344 10824 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:00.124831 10824 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:00.136655 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:00.136703 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:00.136745 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:00.138110 10824 server_base.cc:1061] running on GCE node
I20260812 19:56:00.138797 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:00.139045 10824 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:00.139153 10824 hybrid_clock.cc:648] HybridClock initialized: now 1786564560139128 us; error 0 us; skew 500 ppm
I20260812 19:56:00.139518 10824 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:00.143222 10824 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:00.145087 10824 webserver.cc:533] Webserver started at http://127.25.14.2:41601/ using document root <none> and password file <none>
I20260812 19:56:00.145954 10824 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:00.146059 10824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:00.146366 10824 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:00.148958 10824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "0729c5b385994ea8bea2cbcc03fdf348"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.149870 10824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "0729c5b385994ea8bea2cbcc03fdf348"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.155680 10824 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:56:00.159569 10844 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.161337 10824 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 19:56:00.161550 10824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "0729c5b385994ea8bea2cbcc03fdf348"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.161819 10824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:00.185892 10824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:00.190165 10824 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:00.190500 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:00.191555 10824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:00.193754 10824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:00.193862 10824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.194060 10824 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:00.194124 10824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.232190 10824 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:46157
I20260812 19:56:00.232295 10985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:46157 every 8 connection(s)
I20260812 19:56:00.233913 10824 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:00.235678 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10824
I20260812 19:56:00.235946 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:00.283272 10990 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
I20260812 19:56:00.283807 10990 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.284991 10990 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:00.285756 10988 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:00.286003 10988 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.286595 10988 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
I20260812 19:56:00.287238 10439 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:46157)
I20260812 19:56:00.288055 10532 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:46157)
I20260812 19:56:00.288179 10439 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:49893
I20260812 19:56:00.296506 10989 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:00.296720 10989 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.297120 10989 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:00.298491 10345 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:46157)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:00.385671 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:00.613107 10807 heartbeater.cc:499] Master 127.25.14.61:41507 was elected leader, sending a full tablet report...
W20260812 19:56:00.784111 11006 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:00.784513 11006 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:00.784673 11006 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:00.793923 11006 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:00.794191 11006 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:00.806052 11006 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:00.808051 11006 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:00.809598 11006 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:00.821039 11013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:00.822005 11015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:00.822956 11012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:00.823457 11006 server_base.cc:1061] running on GCE node
I20260812 19:56:00.824225 11006 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:00.824432 11006 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:00.824556 11006 hybrid_clock.cc:648] HybridClock initialized: now 1786564560824521 us; error 0 us; skew 500 ppm
I20260812 19:56:00.824904 11006 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:00.828753 11006 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:00.831068 11006 webserver.cc:533] Webserver started at http://127.25.14.3:35901/ using document root <none> and password file <none>
I20260812 19:56:00.831983 11006 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:00.832147 11006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:00.832549 11006 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:00.835395 11006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "8192f7b281d44f7fa22f3d20c931ea0e"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.836328 11006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "8192f7b281d44f7fa22f3d20c931ea0e"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.843405 11006 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 19:56:00.848017 11023 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.850085 11006 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:56:00.850324 11006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "8192f7b281d44f7fa22f3d20c931ea0e"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:00.850590 11006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:00.872850 11006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:00.877482 11006 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:00.877866 11006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:00.879215 11006 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:00.881245 11006 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:00.881364 11006 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.881536 11006 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:00.881619 11006 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:00.919975 11006 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:33247
I20260812 19:56:00.920038 11154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:33247 every 8 connection(s)
I20260812 19:56:00.921584 11006 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:00.930588 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11006
I20260812 19:56:00.930873 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:00 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564560, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:00.951469 11159 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:00.951728 11157 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:00.952044 11156 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:00.951968 11157 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.951967 11159 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.952203 11156 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:00.953523 11156 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:56:00.953557 11159 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:00.953670 11157 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:00.955917 10532 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:33247)
I20260812 19:56:00.955926 10439 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:33247)
I20260812 19:56:00.956048 10345 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:33247)
I20260812 19:56:00.956974 10439 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:36129
I20260812 19:56:00.960402 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:00.968719 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:00.968914 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10309
I20260812 19:56:00.969957 10384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:01.083369 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10309
I20260812 19:56:01.116444 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--webserver_interface=127.25.14.62
--webserver_port=34339
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:01.151539 10421 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 -> Peer 721a195de0494d6a83841e456c57556a (127.25.14.62:33389): Couldn't send request to peer 721a195de0494d6a83841e456c57556a. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:33389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:56:01.290993 10990 heartbeater.cc:499] Master 127.25.14.61:41507 was elected leader, sending a full tablet report...
W20260812 19:56:01.301191 10989 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:33389 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:33389: connect: Connection refused (error 111)
W20260812 19:56:01.502486 11168 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:01.502902 11168 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:01.503077 11168 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:01.512195 11168 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:01.512320 11168 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:01.512478 11168 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:01.512557 11168 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:01.524642 11168 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=34339
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:01.526551 11168 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:01.528004 11168 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:01.536953 11178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:01.537253 11177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:01.537863 11180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:01.538551 11168 server_base.cc:1061] running on GCE node
I20260812 19:56:01.539508 11168 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:01.539687 11168 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:01.539808 11168 hybrid_clock.cc:648] HybridClock initialized: now 1786564561539769 us; error 0 us; skew 500 ppm
I20260812 19:56:01.540205 11168 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:01 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:01.545585 11168 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:01.552027 11168 webserver.cc:533] Webserver started at http://127.25.14.62:34339/ using document root <none> and password file <none>
I20260812 19:56:01.553295 11168 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:01.553498 11168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:01.562976 11168 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 19:56:01.568045 11187 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:01.570904 11168 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:56:01.571305 11168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:56:01.572695 11168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:01.592981 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:01.593384 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:01.614874 10806 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:33389 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:33389: connect: Connection refused (error 111)
I20260812 19:56:01.648985 11168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:01.654556 11168 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:01.654822 11168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:01.678272 11168 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33389
I20260812 19:56:01.678287 11250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33389 every 8 connection(s)
I20260812 19:56:01.680114 11168 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:01.682176 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11168
I20260812 19:56:01.682552 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10647
I20260812 19:56:01.690395 11252 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:56:01 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:01.694555 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap starting.
I20260812 19:56:01.702814 10800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:01.715437 11252 log.cc:826] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:01.724911 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:01.725536 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap complete.
I20260812 19:56:01.733119 11252 raft_consensus.cc:359] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:01.734007 11252 raft_consensus.cc:740] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721a195de0494d6a83841e456c57556a, State: Initialized, Role: FOLLOWER
I20260812 19:56:01.735025 11252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:01.736022 11260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:01.736377 11260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:01.737377 11252 sys_catalog.cc:565] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:01.753417 11275 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:01.755692 11275 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: loading cluster ID for follower catalog manager: success
I20260812 19:56:01.760969 11275 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: acquiring CA information for follower catalog manager: success
I20260812 19:56:01.764531 11275 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:56:01 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:01.810381 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10647
I20260812 19:56:01.835529 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39527
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=34627
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:01.959589 11159 heartbeater.cc:499] Master 127.25.14.61:41507 was elected leader, sending a full tablet report...
I20260812 19:56:01.970220 11212 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8192f7b281d44f7fa22f3d20c931ea0e" instance_seqno: 1786564560910484) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:49963; Asking this server to re-register.
I20260812 19:56:01.971271 11157 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:01.973418 11212 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:33247)
I20260812 19:56:02.194078 11260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:02.194425 11260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:02.229846 11283 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:02.230196 11283 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:02.230402 11283 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:02.240741 11283 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:02.240967 11283 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:02.252179 11283 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34627
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:02.254093 11283 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:02.255631 11283 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:02.267158 11291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:02.267158 11290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:02.267204 11293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:02.268272 11283 server_base.cc:1061] running on GCE node
I20260812 19:56:02.268993 11283 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:02.269187 11283 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:02.269294 11283 hybrid_clock.cc:648] HybridClock initialized: now 1786564562269268 us; error 0 us; skew 500 ppm
I20260812 19:56:02.269712 11283 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:02 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:02.273504 11283 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:02.275456 11283 webserver.cc:533] Webserver started at http://127.25.14.1:34627/ using document root <none> and password file <none>
I20260812 19:56:02.276518 11283 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:02.276674 11283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:02.284713 11283 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 19:56:02.288820 11302 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:02.290603 11283 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 19:56:02.290825 11283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:02.291604 11283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:02.309464 10989 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:02.310743 11212 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0729c5b385994ea8bea2cbcc03fdf348" instance_seqno: 1786564560222967) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:37573; Asking this server to re-register.
I20260812 19:56:02.311367 10989 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:02.312593 11212 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:46157)
I20260812 19:56:02.328682 11283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:02.333206 11283 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:02.333467 11283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:02.334563 11283 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:02.336367 11283 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:02.336499 11283 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:02.336619 11283 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:02.336701 11283 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:02.375186 11283 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39527
I20260812 19:56:02.375320 11438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39527 every 8 connection(s)
I20260812 19:56:02.376931 11283 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:02.378707 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11283
Aug 12 19:56:02 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:02 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:02 dist-test-slave-p3t5 krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:02.419098 11440 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:02.419235 11439 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:02.419570 11439 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:02.420094 11441 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:02.420255 11441 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:02.420852 11439 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:56:02.420897 11441 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:02.419571 11440 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:02.426115 11440 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:02.427676 11212 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
I20260812 19:56:02.427721 10532 ts_manager.cc:194] Re-registered known tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
I20260812 19:56:02.430583 10439 ts_manager.cc:194] Re-registered known tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:39527)
I20260812 19:56:02.431536 10439 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:54555
I20260812 19:56:02.450927 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11283
W20260812 19:56:02.451576 11453 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:02.447379 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.25.14.1:39527)
0812 19:56:02.448262 (+ 883us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:02.448269 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:02.448299 (+ 30us) server_negotiation.cc:408] Connection header received
0812 19:56:02.448368 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:02.448375 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:02.448606 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:02.448848 (+ 242us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:02.450326 (+ 1478us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46780: BlockingRecv error: recv got EOF from 127.0.0.1:46780 (error 108)
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":522,"threads_started":1}
W20260812 19:56:02.453157 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11283: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:39527: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:02.453263 11453 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:02.451741 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.25.14.1:39527)
0812 19:56:02.451970 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:02.451975 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:02.451996 (+ 21us) server_negotiation.cc:408] Connection header received
0812 19:56:02.452067 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:02.452074 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:02.452169 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:02.452282 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:02.453002 (+ 720us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46782: BlockingRecv error: recv got EOF from 127.0.0.1:46782 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":506,"threads_started":1}
I20260812 19:56:02.453382 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11283
I20260812 19:56:02.481182 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39527
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=34627
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:02.607194 11193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:02.607578 11193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:02.869999 11455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:02.870400 11455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:02.870573 11455 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:02.870668 11455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:02.879539 11455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:02.879908 11455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:02.891516 11455 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34627
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:02.893327 11455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:02.894833 11455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:02.905516 11464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:02.905757 11465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:02.906555 11467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:02.906955 11455 server_base.cc:1061] running on GCE node
I20260812 19:56:02.907652 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:02.907829 11455 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:02.907948 11455 hybrid_clock.cc:648] HybridClock initialized: now 1786564562907911 us; error 0 us; skew 500 ppm
I20260812 19:56:02.908283 11455 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:02 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:02.912177 11455 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:02.914157 11455 webserver.cc:533] Webserver started at http://127.25.14.1:34627/ using document root <none> and password file <none>
I20260812 19:56:02.915014 11455 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:02.915164 11455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:02.921981 11455 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:56:02.925348 11476 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:02.927074 11455 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:02.927294 11455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:02.928107 11455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:02.940400 11455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:02.944398 11455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:02.944708 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:02.945776 11455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:02.947541 11455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:02.947650 11455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:02.947804 11455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:02.947888 11455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:02.948282 11455 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:02.948371 11455 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:02.948468 11455 fault_injection.cc:59] injecting fault for kudu (pid 11455): FLAGS_fault_before_start (process will exit)
I20260812 19:56:02.972446 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39527
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=34627
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:03.345307 11487 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:03.345799 11487 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:03.345981 11487 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:03.354671 11487 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:03.354940 11487 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:03.366217 11487 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34627
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:03.368227 11487 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:03.369801 11487 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:03.381297 11495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:03.381297 11494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:03.382371 11497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:03.382959 11487 server_base.cc:1061] running on GCE node
I20260812 19:56:03.383702 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:03.383908 11487 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:03.384011 11487 hybrid_clock.cc:648] HybridClock initialized: now 1786564563383970 us; error 0 us; skew 500 ppm
I20260812 19:56:03.384359 11487 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:03 dist-test-slave-p3t5 krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564563, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:03.388159 11487 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:03.390070 11487 webserver.cc:533] Webserver started at http://127.25.14.1:34627/ using document root <none> and password file <none>
I20260812 19:56:03.390919 11487 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:03.391057 11487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:03.397467 11487 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:03.400827 11508 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:03.402704 11487 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:03.402910 11487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:03.403971 11487 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:03.431329 11487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:03.440452 11487 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:03.440783 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:03.442049 11487 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:03.443917 11487 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:03.444021 11487 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:03.444186 11487 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:03.444270 11487 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:03.482708 11487 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39527
I20260812 19:56:03.482848 11655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39527 every 8 connection(s)
I20260812 19:56:03.484524 11487 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:03.494264 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11487
I20260812 19:56:03.494756 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11487
W20260812 19:56:03.500394 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11487: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:39527: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:03.500622 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11487
I20260812 19:56:03.532725 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10824
W20260812 19:56:03.533564 11507 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:03.488748 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:34599 (local address 127.25.14.62:33389)
0812 19:56:03.489818 (+ 1070us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:03.489825 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:03.490693 (+ 868us) server_negotiation.cc:408] Connection header received
0812 19:56:03.493357 (+ 2664us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:03.493364 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:03.493480 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:03.493567 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:03.497588 (+ 4021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:03.499049 (+ 1461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:03.532584 (+ 33535us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:34599: BlockingRecv error: recv got EOF from 127.25.14.1:34599 (error 108)
Metrics: {"server-negotiator.queue_time_us":895}
W20260812 19:56:03.533587 11666 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:03.491747 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:50655 (local address 127.25.14.60:45069)
0812 19:56:03.492374 (+ 627us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:03.492380 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:03.492937 (+ 557us) server_negotiation.cc:408] Connection header received
0812 19:56:03.493748 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:03.493755 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:03.493868 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:03.494009 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:03.502760 (+ 8751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:03.504218 (+ 1458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:03.532480 (+ 28262us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:50655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
W20260812 19:56:03.533593 11660 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:03.488400 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:51341 (local address 127.25.14.61:41507)
0812 19:56:03.491872 (+ 3472us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:03.491879 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:03.491906 (+ 27us) server_negotiation.cc:408] Connection header received
0812 19:56:03.493527 (+ 1621us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:03.493532 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:03.493671 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:03.493810 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:03.497497 (+ 3687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:03.499049 (+ 1552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:03.500717 (+ 1668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:03.511323 (+ 10606us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:56:03.532617 (+ 21294us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:51341: BlockingRecv error: recv got EOF from 127.25.14.1:51341 (error 108)
Metrics: {"server-negotiator.queue_time_us":3343,"thread_start_us":3214,"threads_started":1}
W20260812 19:56:03.536247 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10824: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:46157: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:03.536464 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10824
I20260812 19:56:03.560706 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11006
W20260812 19:56:03.563705 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11006: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:33247: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:03.563875 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11006
I20260812 19:56:03.589295 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11168
W20260812 19:56:03.591662 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11168: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:33389: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:03.591822 11507 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:03.590416 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.25.14.62:33389)
0812 19:56:03.590595 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:03.590600 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:03.590620 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:56:03.590713 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:03.590718 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:03.590786 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:03.590879 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:03.591537 (+ 658us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41588: BlockingRecv error: recv got EOF from 127.0.0.1:41588 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 19:56:03.591949 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11168
I20260812 19:56:03.624384 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10403
I20260812 19:56:03.625798 10474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:03.741704 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10403
I20260812 19:56:03.770762 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10493
I20260812 19:56:03.772198 10568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:03.886973 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10493
Aug 12 19:56:03 dist-test-slave-p3t5 krb5kdc[11677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:03 dist-test-slave-p3t5 krb5kdc[11677](info): set up 2 sockets
Aug 12 19:56:03 dist-test-slave-p3t5 krb5kdc[11677](info): commencing operation
krb5kdc: starting...
W20260812 19:56:06.031051 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.109s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:06 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:06.316913 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:06.695345 11694 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:06.695708 11694 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:06.695804 11694 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:06.704504 11694 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:06.704629 11694 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:06.704692 11694 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:06.704741 11694 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:06.715893 11694 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:06.717797 11694 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:06.719278 11694 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:06.728904 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:06.728866 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:06.729337 11706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:06.730319 11694 server_base.cc:1061] running on GCE node
I20260812 19:56:06.731200 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:06.731356 11694 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:06.731428 11694 hybrid_clock.cc:648] HybridClock initialized: now 1786564566731409 us; error 0 us; skew 500 ppm
I20260812 19:56:06.731730 11694 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:06 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:06.735652 11694 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:06.737982 11694 webserver.cc:533] Webserver started at http://127.25.14.62:41901/ using document root <none> and password file <none>
I20260812 19:56:06.738778 11694 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:06.738938 11694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:06.745208 11694 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 19:56:06.748548 11714 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:06.750151 11694 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:56:06.750361 11694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:56:06.751168 11694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:06.797947 11694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:06.802554 11694 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:06.802887 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:06.825848 11694 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33389
I20260812 19:56:06.825896 11780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33389 every 8 connection(s)
I20260812 19:56:06.827688 11694 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:06.830021 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11694
I20260812 19:56:06.830259 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:56:06.835855 11781 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:06.839723 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap starting.
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:06.858343 11781 log.cc:826] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:06.868134 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:06.868738 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap complete.
I20260812 19:56:06.880824 11781 raft_consensus.cc:359] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:06.882097 11781 raft_consensus.cc:740] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721a195de0494d6a83841e456c57556a, State: Initialized, Role: FOLLOWER
I20260812 19:56:06.883219 11781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:06.884280 11788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:06.884596 11788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:06.885943 11781 sys_catalog.cc:565] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:06.904902 11801 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:06.905118 11801 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: loading cluster ID for follower catalog manager: success
Aug 12 19:56:06 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:06.910112 11801 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: acquiring CA information for follower catalog manager: success
I20260812 19:56:06.915423 11801 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:06.924641 11694 master_runner.cc:187] Error getting master registration for 127.25.14.61:41507: Network error: Client connection negotiation failed: client connection to 127.25.14.61:41507: connect: Connection refused (error 111),
I20260812 19:56:06.931564 11694 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:06.969810 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41507
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:07.347803 11807 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:07.348230 11807 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:07.348400 11807 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:07.356817 11807 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:07.357005 11807 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:07.357087 11807 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:07.357136 11807 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:07.368198 11807 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:07.369975 11807 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:07.371400 11807 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:07.380541 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:07.380669 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:07.381489 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:07.382179 11807 server_base.cc:1061] running on GCE node
I20260812 19:56:07.383216 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:07.383441 11807 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:07.383555 11807 hybrid_clock.cc:648] HybridClock initialized: now 1786564567383512 us; error 0 us; skew 500 ppm
I20260812 19:56:07.383972 11807 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:07 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564567, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:07.388209 11807 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:56:07.390720 11807 webserver.cc:533] Webserver started at http://127.25.14.61:34465/ using document root <none> and password file <none>
I20260812 19:56:07.391558 11807 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:07.391688 11807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:07.398077 11807 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:07.401134 11822 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:07.402721 11807 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:07.402963 11807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "4d385919465b42b3b0ad831bcdb77f91"
format_stamp: "Formatted at 2026-08-12 19:55:57 on dist-test-slave-p3t5"
I20260812 19:56:07.403739 11807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:07.431089 11807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:07.435617 11807 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:07.435933 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:07.458796 11807 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:41507
I20260812 19:56:07.458815 11883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:41507 every 8 connection(s)
I20260812 19:56:07.460672 11807 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:56:07.468379 11884 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:07.471084 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11807
I20260812 19:56:07.471298 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:56:07.472774 11884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Bootstrap starting.
I20260812 19:56:07.494002 11884 log.cc:826] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:07.503983 11884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:07.504601 11884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Bootstrap complete.
I20260812 19:56:07.512099 11884 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:07.512929 11884 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d385919465b42b3b0ad831bcdb77f91, State: Initialized, Role: FOLLOWER
I20260812 19:56:07.513907 11884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:07.514941 11891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:07.515265 11891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:07.516244 11884 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:07.528461 11902 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:07.528719 11902 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: loading cluster ID for follower catalog manager: success
I20260812 19:56:07.534413 11902 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: acquiring CA information for follower catalog manager: success
I20260812 19:56:07.537590 11902 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:07.564107 11807 master_runner.cc:187] Error getting master registration for 127.25.14.60:45069: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45069: connect: Connection refused (error 111),
I20260812 19:56:07.568145 11807 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:07.621315 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45069
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:08.038020 11909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:08.038415 11909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:08.038582 11909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:08.047305 11909 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:08.047421 11909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:08.047530 11909 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:08.047617 11909 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:08.059012 11909 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:08.061025 11909 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:08.062584 11909 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:08.071590 11916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:08.071683 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:08.072057 11918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:08.072791 11909 server_base.cc:1061] running on GCE node
I20260812 19:56:08.073698 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:08.073860 11909 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:08.073935 11909 hybrid_clock.cc:648] HybridClock initialized: now 1786564568073915 us; error 0 us; skew 500 ppm
I20260812 19:56:08.074239 11909 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:08 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:08.078248 11909 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:56:08.080482 11909 webserver.cc:533] Webserver started at http://127.25.14.60:45905/ using document root <none> and password file <none>
I20260812 19:56:08.081306 11909 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:08.081492 11909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:08.087625 11909 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:56:08.090976 11927 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:08.092499 11909 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:08.092700 11909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
format_stamp: "Formatted at 2026-08-12 19:55:58 on dist-test-slave-p3t5"
I20260812 19:56:08.093493 11909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:08.112099 11909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:08.116428 11909 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:08.116751 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:08.139802 11909 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:45069
I20260812 19:56:08.139838 11988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:45069 every 8 connection(s)
I20260812 19:56:08.141569 11909 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:56:08.148859 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11909
I20260812 19:56:08.149097 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:56:08.149184 11989 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:08.154527 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Bootstrap starting.
Aug 12 19:56:08 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:08.175817 11989 log.cc:826] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Log is configured to *not* fsync() on all Append() calls
Aug 12 19:56:08 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:08.187860 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:08.188555 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: Bootstrap complete.
I20260812 19:56:08.195761 11989 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:08.196557 11989 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a4dfd879ee4b95b2f75d60c4249a92, State: Initialized, Role: FOLLOWER
I20260812 19:56:08.197430 11989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
Aug 12 19:56:08 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:08.198570 12000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.198969 12000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:08.199910 11989 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:08.211901 12014 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:08.212018 12014 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: loading cluster ID for follower catalog manager: success
I20260812 19:56:08.216275 12014 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: acquiring CA information for follower catalog manager: success
I20260812 19:56:08.219313 12014 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:08.246451 11909 master_runner.cc:264] No leader master found from master 69a4dfd879ee4b95b2f75d60c4249a92
I20260812 19:56:08.246591 11909 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:08.348847 12020 raft_consensus.cc:493] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:56:08.349278 12020 raft_consensus.cc:515] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:08.353056 12020 leader_election.cc:290] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d385919465b42b3b0ad831bcdb77f91 (127.25.14.61:41507), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:56:08.360765 11852 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "721a195de0494d6a83841e456c57556a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d385919465b42b3b0ad831bcdb77f91" is_pre_election: true
I20260812 19:56:08.361390 11852 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721a195de0494d6a83841e456c57556a in term 1.
I20260812 19:56:08.361704 11960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "721a195de0494d6a83841e456c57556a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" is_pre_election: true
I20260812 19:56:08.362146 11719 leader_election.cc:304] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters:
I20260812 19:56:08.362409 11960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721a195de0494d6a83841e456c57556a in term 1.
I20260812 19:56:08.362639 12020 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:56:08.362815 12020 raft_consensus.cc:493] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:56:08.362951 12020 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:08.366888 12020 raft_consensus.cc:515] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:08.367635 12020 leader_election.cc:290] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [CANDIDATE]: Term 2 election: Requested vote from peers 4d385919465b42b3b0ad831bcdb77f91 (127.25.14.61:41507), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:56:08.368089 11852 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "721a195de0494d6a83841e456c57556a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d385919465b42b3b0ad831bcdb77f91"
I20260812 19:56:08.368256 11960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "721a195de0494d6a83841e456c57556a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
I20260812 19:56:08.368323 11852 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:08.368482 11960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:08.373570 11852 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721a195de0494d6a83841e456c57556a in term 2.
I20260812 19:56:08.375296 11719 leader_election.cc:304] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters:
I20260812 19:56:08.375694 12020 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:56:08.375703 11960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721a195de0494d6a83841e456c57556a in term 2.
I20260812 19:56:08.376305 12020 raft_consensus.cc:697] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 LEADER]: Becoming Leader. State: Replica: 721a195de0494d6a83841e456c57556a, State: Running, Role: LEADER
I20260812 19:56:08.376886 12020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:08.379580 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721a195de0494d6a83841e456c57556a. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.379884 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:08.380863 12026 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:08.383664 12026 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:08.384754 12026 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:08.384862 12026 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:08.386464 12026 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:08.387583 12026 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Loaded TSK: 0
I20260812 19:56:08.388859 12026 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:08.519392 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:08.822721 11852 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Refusing update from remote peer 721a195de0494d6a83841e456c57556a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:08.824035 12024 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:08.828078 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721a195de0494d6a83841e456c57556a. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.828482 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:08.842033 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.842319 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:08.843003 12020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.843300 12020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:08.843967 11960 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 2 FOLLOWER]: Refusing update from remote peer 721a195de0494d6a83841e456c57556a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:08.844975 12020 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:08.853922 12043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.854277 12043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:08.854498 12040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721a195de0494d6a83841e456c57556a. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:08.854802 12040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:08.918691 12035 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:08.919036 12035 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:08.919139 12035 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:08.927932 12035 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:08.928148 12035 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:08.939905 12035 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:08.941901 12035 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:08.943444 12035 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:08.954829 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:08.954846 12049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:08.955585 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:08.956378 12035 server_base.cc:1061] running on GCE node
I20260812 19:56:08.957057 12035 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:08.957261 12035 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:08.957424 12035 hybrid_clock.cc:648] HybridClock initialized: now 1786564568957398 us; error 0 us; skew 500 ppm
I20260812 19:56:08.957850 12035 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:08 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:08.961875 12035 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:08.963722 12035 webserver.cc:533] Webserver started at http://127.25.14.1:46365/ using document root <none> and password file <none>
I20260812 19:56:08.964586 12035 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:08.964737 12035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:08.970741 12035 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:08.973850 12058 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:08.975315 12035 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:08.975517 12035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:08.976270 12035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:08.998399 12035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:09.007650 12035 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:09.008059 12035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:09.009363 12035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:09.011277 12035 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:09.011407 12035 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:09.011554 12035 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:09.011658 12035 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:09.050100 12035 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35189
I20260812 19:56:09.050200 12189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35189 every 8 connection(s)
I20260812 19:56:09.051849 12035 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:09.053879 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12035
I20260812 19:56:09.054103 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564568, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:09.094856 12192 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:09.094964 12190 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:09.095139 12193 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:09.095465 12190 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.095397 12192 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.095394 12193 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.096902 12190 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:56:09.096974 12192 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:09.097002 12193 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:09.099888 11734 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:09.099888 11949 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:09.099902 11840 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:09.102317 11734 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:55817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:09.193281 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:09.228786 12020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "721a195de0494d6a83841e456c57556a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:09.229199 12020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:09.570250 12206 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:09.570662 12206 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:09.570835 12206 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:09.579630 12206 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:09.579828 12206 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:09.591060 12206 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:09.592935 12206 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:09.594641 12206 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:09.605098 12213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:09.605815 12214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:09.605847 12216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:09.606470 12206 server_base.cc:1061] running on GCE node
I20260812 19:56:09.607098 12206 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:09.607300 12206 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:09.607380 12206 hybrid_clock.cc:648] HybridClock initialized: now 1786564569607360 us; error 0 us; skew 500 ppm
I20260812 19:56:09.607653 12206 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564569, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:09.611528 12206 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:09.613286 12206 webserver.cc:533] Webserver started at http://127.25.14.2:36733/ using document root <none> and password file <none>
I20260812 19:56:09.614169 12206 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:09.614322 12206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:09.620589 12206 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 19:56:09.624115 12222 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:09.625746 12206 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:09.625923 12206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "0729c5b385994ea8bea2cbcc03fdf348"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:09.626663 12206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:09.651762 12206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:09.655849 12206 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:09.656119 12206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:09.657253 12206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:09.659094 12206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:09.659216 12206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:09.659369 12206 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:09.659458 12206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:09.696107 12206 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:37769
I20260812 19:56:09.696173 12361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:37769 every 8 connection(s)
I20260812 19:56:09.697892 12206 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:09.706177 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12206
I20260812 19:56:09.706478 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564569, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564569, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:09 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564569, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:09.748104 12363 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:09.748206 12362 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:09.748138 12364 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:09.748639 12363 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.748812 12364 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.748636 12362 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:09.750254 12363 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:09.750283 12364 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:09.750310 12362 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:09.752401 11734 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:37769)
I20260812 19:56:09.752506 11840 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:37769)
I20260812 19:56:09.752595 11949 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:37769)
I20260812 19:56:09.753510 11734 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:60169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:09.854094 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:10.105089 12192 heartbeater.cc:499] Master 127.25.14.62:33389 was elected leader, sending a full tablet report...
W20260812 19:56:10.236266 12378 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:10.236687 12378 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:10.236861 12378 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:10.245599 12378 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:10.246018 12378 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:10.257824 12378 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:10.259703 12378 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:10.261212 12378 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:10.272491 12385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:10.272576 12389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:10.272549 12386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:10.274149 12378 server_base.cc:1061] running on GCE node
I20260812 19:56:10.274803 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:10.275013 12378 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:10.275117 12378 hybrid_clock.cc:648] HybridClock initialized: now 1786564570275096 us; error 0 us; skew 500 ppm
I20260812 19:56:10.275421 12378 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:10 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:10.279560 12378 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:10.281752 12378 webserver.cc:533] Webserver started at http://127.25.14.3:41283/ using document root <none> and password file <none>
I20260812 19:56:10.282653 12378 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:10.282805 12378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:10.289266 12378 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:56:10.292793 12396 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:10.294536 12378 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:10.294746 12378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "8192f7b281d44f7fa22f3d20c931ea0e"
format_stamp: "Formatted at 2026-08-12 19:56:00 on dist-test-slave-p3t5"
I20260812 19:56:10.295557 12378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:10.328200 12378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:10.332893 12378 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:10.333233 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:10.334357 12378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:10.336141 12378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:10.336278 12378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:10.336421 12378 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:10.336519 12378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:10.374639 12378 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:46551
I20260812 19:56:10.374706 12534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:46551 every 8 connection(s)
I20260812 19:56:10.376322 12378 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:10.386670 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12378
I20260812 19:56:10.386909 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:10 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:10 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:10 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564570, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:10.408012 12536 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:10.408089 12535 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:10.408536 12536 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:10.408545 12535 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:10.408986 12537 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:10.409160 12537 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:10.410015 12537 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:10.410032 12535 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:56:10.411202 12536 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:10.412307 11948 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:46551)
I20260812 19:56:10.412577 11840 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:46551)
I20260812 19:56:10.413278 11734 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:46551)
I20260812 19:56:10.414333 11734 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:46503
I20260812 19:56:10.432343 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:10.442821 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:10.443130 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11694
I20260812 19:56:10.444218 11775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:10.559629 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11694
W20260812 19:56:10.592103 11931 connection.cc:573] server connection from 127.0.0.1:52718 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:56:10.593426 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--webserver_interface=127.25.14.62
--webserver_port=41901
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:10.756651 12363 heartbeater.cc:499] Master 127.25.14.62:33389 was elected leader, sending a full tablet report...
W20260812 19:56:10.758618 12363 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:33389 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:33389: connect: Connection refused (error 111)
W20260812 19:56:10.990484 12546 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:10.991039 12546 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:10.991238 12546 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:11.000677 12546 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:11.000855 12546 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:11.000927 12546 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:11.001011 12546 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:11.012800 12546 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=41901
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:11.014932 12546 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:11.016589 12546 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:11.026918 12558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:11.027047 12554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:11.027019 12555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:11.028026 12546 server_base.cc:1061] running on GCE node
I20260812 19:56:11.028828 12546 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:11.028992 12546 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:11.029074 12546 hybrid_clock.cc:648] HybridClock initialized: now 1786564571029056 us; error 0 us; skew 500 ppm
I20260812 19:56:11.029381 12546 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:11.033952 12546 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:11.036242 12546 webserver.cc:533] Webserver started at http://127.25.14.62:41901/ using document root <none> and password file <none>
I20260812 19:56:11.037034 12546 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:11.037137 12546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:11.043437 12546 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:56:11.046967 12567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:11.048530 12546 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:11.048743 12546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "721a195de0494d6a83841e456c57556a"
format_stamp: "Formatted at 2026-08-12 19:55:56 on dist-test-slave-p3t5"
I20260812 19:56:11.049571 12546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:11.068907 12546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:11.073823 12546 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:11.074150 12546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:11.096710 12546 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:33389
I20260812 19:56:11.096730 12631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:33389 every 8 connection(s)
I20260812 19:56:11.098614 12546 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:11.106494 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12546
I20260812 19:56:11.106909 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12035
I20260812 19:56:11.107252 12633 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:11.119961 12633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap starting.
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:11.142961 12185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:11.143800 12633 log.cc:826] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:11.162475 12633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:11.163300 12633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: Bootstrap complete.
I20260812 19:56:11.173708 12633 raft_consensus.cc:359] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:11.174774 12633 raft_consensus.cc:740] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721a195de0494d6a83841e456c57556a, State: Initialized, Role: FOLLOWER
I20260812 19:56:11.175680 12633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:11.176653 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:11.176977 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:11.178172 12633 sys_catalog.cc:565] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:11.195803 12664 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:11.195942 12664 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: loading cluster ID for follower catalog manager: success
I20260812 19:56:11.199218 12664 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: acquiring CA information for follower catalog manager: success
I20260812 19:56:11.202294 12664 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:11.231794 12546 master_runner.cc:264] No leader master found from master 721a195de0494d6a83841e456c57556a
I20260812 19:56:11.231962 12546 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:56:11.277503 12192 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:33389 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 19:56:11.278920 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12035
I20260812 19:56:11.304344 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35189
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=46365
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:11.417516 12536 heartbeater.cc:499] Master 127.25.14.62:33389 was elected leader, sending a full tablet report...
I20260812 19:56:11.428455 12586 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8192f7b281d44f7fa22f3d20c931ea0e" instance_seqno: 1786564570365167) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:34245; Asking this server to re-register.
I20260812 19:56:11.429451 12536 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:11.431445 12586 ts_manager.cc:194] Registered new tserver with Master: 8192f7b281d44f7fa22f3d20c931ea0e (127.25.14.3:46551)
W20260812 19:56:11.711620 12671 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:11.712128 12671 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:11.712302 12671 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:11.724143 12671 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:11.724429 12671 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:11.742028 12671 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46365
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:11.744004 12671 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:11.746420 12671 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:11.758563 12679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:11.759668 12680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:11.760596 12682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:11.761937 12671 server_base.cc:1061] running on GCE node
I20260812 19:56:11.762831 12671 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:11.763060 12671 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:11.763163 12671 hybrid_clock.cc:648] HybridClock initialized: now 1786564571763140 us; error 0 us; skew 500 ppm
I20260812 19:56:11.763559 12671 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:11.768817 12363 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:11.768899 12671 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:11.769060 12363 heartbeater.cc:499] Master 127.25.14.62:33389 was elected leader, sending a full tablet report...
I20260812 19:56:11.770186 12586 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0729c5b385994ea8bea2cbcc03fdf348" instance_seqno: 1786564569687055) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:60839; Asking this server to re-register.
I20260812 19:56:11.770893 12363 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:11.771781 12671 webserver.cc:533] Webserver started at http://127.25.14.1:46365/ using document root <none> and password file <none>
I20260812 19:56:11.772229 12586 ts_manager.cc:194] Registered new tserver with Master: 0729c5b385994ea8bea2cbcc03fdf348 (127.25.14.2:37769)
I20260812 19:56:11.772807 12671 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:11.772964 12671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:11.779381 12671 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:56:11.782836 12692 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:11.784698 12671 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:11.784986 12671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:11.785909 12671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:11.834460 12671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:11.838739 12671 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:11.839064 12671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:11.840181 12671 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:11.842170 12671 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:11.842305 12671 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:11.842446 12671 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:11.842545 12671 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:11.879678 12671 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35189
I20260812 19:56:11.879776 12833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35189 every 8 connection(s)
I20260812 19:56:11.881556 12671 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:11.889647 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12671
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:11 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564571, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:11.919562 12834 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45069
I20260812 19:56:11.920024 12834 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:11.921362 12834 heartbeater.cc:507] Master 127.25.14.60:45069 requested a full tablet report, sending...
I20260812 19:56:11.923053 12835 heartbeater.cc:344] Connected to a master server at 127.25.14.62:33389
I20260812 19:56:11.923235 12835 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:11.923791 12835 heartbeater.cc:507] Master 127.25.14.62:33389 requested a full tablet report, sending...
I20260812 19:56:11.924088 11948 ts_manager.cc:194] Re-registered known tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:11.924998 12586 ts_manager.cc:194] Registered new tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:11.931334 12847 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721a195de0494d6a83841e456c57556a)
I20260812 19:56:11.931619 12847 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:11.932327 12836 heartbeater.cc:344] Connected to a master server at 127.25.14.61:41507
I20260812 19:56:11.932549 12836 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:11.933161 12836 heartbeater.cc:507] Master 127.25.14.61:41507 requested a full tablet report, sending...
I20260812 19:56:11.934239 12847 leader_election.cc:290] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721a195de0494d6a83841e456c57556a (127.25.14.62:33389), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:56:11.935467 11840 ts_manager.cc:194] Re-registered known tserver with Master: 1f585003c77542d099216a746029ffab (127.25.14.1:35189)
I20260812 19:56:11.942836 11960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" is_pre_election: true
I20260812 19:56:11.945154 12601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721a195de0494d6a83841e456c57556a" is_pre_election: true
I20260812 19:56:11.945878 12601 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 2.
I20260812 19:56:11.946578 11824 leader_election.cc:304] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters: 69a4dfd879ee4b95b2f75d60c4249a92
I20260812 19:56:11.947131 12847 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:56:11.947279 12847 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721a195de0494d6a83841e456c57556a)
I20260812 19:56:11.947412 12847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:11.949687 12847 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:11.950443 12847 leader_election.cc:290] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 3 election: Requested vote from peers 721a195de0494d6a83841e456c57556a (127.25.14.62:33389), 69a4dfd879ee4b95b2f75d60c4249a92 (127.25.14.60:45069)
I20260812 19:56:11.951143 12601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721a195de0494d6a83841e456c57556a"
I20260812 19:56:11.951241 11960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d385919465b42b3b0ad831bcdb77f91" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69a4dfd879ee4b95b2f75d60c4249a92"
I20260812 19:56:11.951393 12601 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:11.957881 12601 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d385919465b42b3b0ad831bcdb77f91 in term 3.
I20260812 19:56:11.958561 11824 leader_election.cc:304] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a; no voters: 69a4dfd879ee4b95b2f75d60c4249a92
I20260812 19:56:11.959002 12847 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:56:11.959578 12847 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [term 3 LEADER]: Becoming Leader. State: Replica: 4d385919465b42b3b0ad831bcdb77f91, State: Running, Role: LEADER
I20260812 19:56:11.960078 12847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:11.961985 12851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:11.962277 12851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:11.963081 12856 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:11.965482 12856 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:11.966547 12856 catalog_manager.cc:1295] Loaded cluster ID: 34d1cf2ee9464768ace296efc1b9c135
I20260812 19:56:11.966641 12856 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:11.967514 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12671
I20260812 19:56:11.968257 12856 catalog_manager.cc:1540] Loading token signing keys...
W20260812 19:56:11.968572 12859 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:11.963948 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.25.14.1:35189)
0812 19:56:11.965410 (+ 1462us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:11.965417 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:11.965448 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:56:11.965507 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:11.965513 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:11.965841 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:11.966010 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:11.967196 (+ 1186us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40386: BlockingRecv error: recv got EOF from 127.0.0.1:40386 (error 108)
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1161,"threads_started":1}
I20260812 19:56:11.969286 12856 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91: Loaded TSK: 0
I20260812 19:56:11.970228 12856 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260812 19:56:11.970624 12859 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:11.968767 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.25.14.1:35189)
0812 19:56:11.969211 (+ 444us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:11.969217 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:11.969238 (+ 21us) server_negotiation.cc:408] Connection header received
0812 19:56:11.969498 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:11.969504 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:11.969592 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:11.969750 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:11.970402 (+ 652us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40392: BlockingRecv error: recv got EOF from 127.0.0.1:40392 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":707,"threads_started":1}
W20260812 19:56:11.970595 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12671: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:35189: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:11.970786 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12671
I20260812 19:56:11.996546 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35189
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=46365
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:12.062762 12862 raft_consensus.cc:493] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721a195de0494d6a83841e456c57556a)
I20260812 19:56:12.063062 12862 raft_consensus.cc:515] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } }
I20260812 19:56:12.065066 12862 leader_election.cc:290] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721a195de0494d6a83841e456c57556a (127.25.14.62:33389), 4d385919465b42b3b0ad831bcdb77f91 (127.25.14.61:41507)
I20260812 19:56:12.065979 11852 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d385919465b42b3b0ad831bcdb77f91" is_pre_election: true
I20260812 19:56:12.072736 12601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721a195de0494d6a83841e456c57556a" is_pre_election: true
I20260812 19:56:12.073170 12601 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69a4dfd879ee4b95b2f75d60c4249a92 in current term 3: Already voted for candidate 4d385919465b42b3b0ad831bcdb77f91 in this term.
I20260812 19:56:12.073832 11931 leader_election.cc:304] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69a4dfd879ee4b95b2f75d60c4249a92; no voters: 4d385919465b42b3b0ad831bcdb77f91, 721a195de0494d6a83841e456c57556a
I20260812 19:56:12.074375 12862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:12.077652 12862 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 19:56:12.381139 12861 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:12.381502 12861 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:12.381599 12861 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:12.381737 12861 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:12.390792 12861 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:12.391012 12861 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:12.402273 12861 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46365
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:12.404078 12861 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:12.405512 12861 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:12.415989 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:12.417413 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:12.418591 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:12.419540 12861 server_base.cc:1061] running on GCE node
I20260812 19:56:12.420289 12861 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:12.420478 12861 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:12.420607 12861 hybrid_clock.cc:648] HybridClock initialized: now 1786564572420560 us; error 0 us; skew 500 ppm
I20260812 19:56:12.420936 12861 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:12 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:12.424928 12861 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:12.426882 12861 webserver.cc:533] Webserver started at http://127.25.14.1:46365/ using document root <none> and password file <none>
I20260812 19:56:12.427757 12861 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:12.427899 12861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:12.434284 12861 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:12.437386 12878 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:12.439114 12861 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:12.439325 12861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:12.440109 12861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:12.474793 12601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [term 3 FOLLOWER]: Refusing update from remote peer 4d385919465b42b3b0ad831bcdb77f91: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:12.475915 12884 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:12.480489 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:12.481014 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:12.482947 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:12.483203 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:12.483973 12861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:12.484333 11960 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [term 3 FOLLOWER]: Refusing update from remote peer 4d385919465b42b3b0ad831bcdb77f91: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:12.491128 12861 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:12.491632 12861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:12.495318 12884 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:12.497808 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:12.498219 12861 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:12.498322 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721a195de0494d6a83841e456c57556a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:12.500504 12898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:12.500599 12862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d385919465b42b3b0ad831bcdb77f91. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:12.500917 12898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:12.500967 12862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a4dfd879ee4b95b2f75d60c4249a92 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:12.502277 12861 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:12.502365 12861 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:12.502539 12861 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:12.502592 12861 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:12.503010 12861 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:12.503118 12861 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:12.503206 12861 fault_injection.cc:59] injecting fault for kudu (pid 12861): FLAGS_fault_before_start (process will exit)
I20260812 19:56:12.529796 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35189
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--webserver_port=46365
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:12.917004 12902 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:12.917433 12902 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:12.917595 12902 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:12.926652 12902 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:12.926954 12902 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:12.938206 12902 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=46365
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:33389,127.25.14.61:41507,127.25.14.60:45069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:12.940181 12902 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:12.941874 12902 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:12.952793 12908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:12.952793 12907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:12.953286 12910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:12.954210 12902 server_base.cc:1061] running on GCE node
I20260812 19:56:12.954952 12902 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:56:12.955178 12902 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:56:12.955294 12902 hybrid_clock.cc:648] HybridClock initialized: now 1786564572955268 us; error 0 us; skew 500 ppm
I20260812 19:56:12.955644 12902 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:12 dist-test-slave-p3t5 krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:12.959668 12902 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:12.961593 12902 webserver.cc:533] Webserver started at http://127.25.14.1:46365/ using document root <none> and password file <none>
I20260812 19:56:12.962551 12902 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:12.962700 12902 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:12.968677 12902 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 19:56:12.971895 12917 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:12.973395 12902 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:56:12.973588 12902 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "1f585003c77542d099216a746029ffab"
format_stamp: "Formatted at 2026-08-12 19:55:59 on dist-test-slave-p3t5"
I20260812 19:56:12.974599 12902 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:12.999126 12902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:13.004670 12902 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:13.004997 12902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:13.006152 12902 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:13.007967 12902 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:13.008067 12902 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:13.008229 12902 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:13.008354 12902 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:13.044744 12902 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35189
I20260812 19:56:13.044906 13061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35189 every 8 connection(s)
I20260812 19:56:13.046584 12902 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:13.053687 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12902
I20260812 19:56:13.054153 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12902
Aug 12 19:56:13 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:13 dist-test-slave-p3t5 krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:13.079864 13079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4d385919465b42b3b0ad831bcdb77f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721a195de0494d6a83841e456c57556a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 33389 } } peers { permanent_uuid: "4d385919465b42b3b0ad831bcdb77f91" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 41507 } } peers { permanent_uuid: "69a4dfd879ee4b95b2f75d60c4249a92" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45069 } } }
I20260812 19:56:13.080274 13079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d385919465b42b3b0ad831bcdb77f91 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:13.083698 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12902: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:35189: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:13.083969 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12902
I20260812 19:56:13.110839 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12206
W20260812 19:56:13.111888 13067 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:13.050778 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:39595 (local address 127.25.14.61:41507)
0812 19:56:13.051265 (+ 487us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:13.051272 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:13.060894 (+ 9622us) server_negotiation.cc:408] Connection header received
0812 19:56:13.062117 (+ 1223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:13.062126 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:13.062255 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:13.062403 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:13.065024 (+ 2621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:13.067094 (+ 2070us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:13.073522 (+ 6428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:13.074282 (+ 760us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:56:13.079783 (+ 5501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:56:13.079821 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:56:13.079828 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 19:56:13.079876 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:56:13.085863 (+ 5987us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 19:56:13.110790 (+ 24927us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:39595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
W20260812 19:56:13.111889 13074 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:13.054854 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:60569 (local address 127.25.14.62:33389)
0812 19:56:13.055966 (+ 1112us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:13.055975 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:13.056794 (+ 819us) server_negotiation.cc:408] Connection header received
0812 19:56:13.056912 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:13.056921 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:13.057042 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:13.057182 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:13.066040 (+ 8858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:13.068267 (+ 2227us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:13.070947 (+ 2680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:13.071657 (+ 710us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:56:13.080932 (+ 9275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:56:13.080964 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:56:13.080968 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 19:56:13.081003 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:56:13.087402 (+ 6399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 19:56:13.110788 (+ 23386us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:60569: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":877,"threads_started":1}
W20260812 19:56:13.112248 13072 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:13.052395 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48577 (local address 127.25.14.60:45069)
0812 19:56:13.077274 (+ 24879us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:13.077279 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:13.077297 (+ 18us) server_negotiation.cc:408] Connection header received
0812 19:56:13.077352 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:13.077357 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:13.077442 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:13.077598 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:13.079694 (+ 2096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:13.094873 (+ 15179us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:13.111381 (+ 16508us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":24771,"thread_start_us":24598,"threads_started":1}
W20260812 19:56:13.114383 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12206: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:37769: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:13.114542 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12206
I20260812 19:56:13.140882 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12378
W20260812 19:56:13.144160 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12378: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:46551: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:13.144357 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12378
I20260812 19:56:13.172636 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12546
W20260812 19:56:13.174938 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12546: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:33389: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:13.175020 13074 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:13.173576 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.25.14.62:33389)
0812 19:56:13.173781 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:13.173786 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:13.173805 (+ 19us) server_negotiation.cc:408] Connection header received
0812 19:56:13.173984 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:13.173990 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:13.174068 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:13.174161 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:13.174826 (+ 665us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48756: BlockingRecv error: recv got EOF from 127.0.0.1:48756 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 19:56:13.175153 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12546
I20260812 19:56:13.205763 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11807
I20260812 19:56:13.207089 11879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:13.248611 11909 master_runner.cc:187] Error getting master registration for 127.25.14.62:33389: Network error: Client connection negotiation failed: client connection to 127.25.14.62:33389: connect: Connection refused (error 111),
I20260812 19:56:13.252351 11909 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:13.324388 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11807
I20260812 19:56:13.354516 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11909
I20260812 19:56:13.355809 11983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:13.468976 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11909
[ OK ] ExternalMiniClusterTest.TestBasicOperation/6 (19492 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/7
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:56:13 dist-test-slave-p3t5 krb5kdc[13090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:13 dist-test-slave-p3t5 krb5kdc[13090](info): set up 2 sockets
Aug 12 19:56:13 dist-test-slave-p3t5 krb5kdc[13090](info): commencing operation
krb5kdc: starting...
W20260812 19:56:15.668351 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.156s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:15 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:56:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:16.045737 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:16.417646 13109 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:16.418012 13109 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:16.418167 13109 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:16.426700 13109 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:16.426841 13109 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:16.426949 13109 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:16.427060 13109 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:16.438269 13109 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:16.440068 13109 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:16.441598 13109 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:16.450624 13114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:16.450601 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:16.451612 13118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:16.451799 13109 server_base.cc:1061] running on GCE node
I20260812 19:56:16.452768 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:16.454511 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:16.455730 13109 hybrid_clock.cc:648] HybridClock initialized: now 1786564576455686 us; error 39 us; skew 500 ppm
I20260812 19:56:16.456161 13109 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:16 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:16.459682 13109 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:16.461508 13109 webserver.cc:533] Webserver started at http://127.25.14.62:46841/ using document root <none> and password file <none>
I20260812 19:56:16.462368 13109 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:16.462472 13109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:16.462787 13109 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:16.465327 13109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:16.466295 13109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:16.472415 13109 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:56:16.477002 13126 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:16.478781 13109 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:16.479015 13109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:16.479256 13109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:16.522732 13109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:16.533110 13109 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:16.533456 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:16.556021 13109 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41219
I20260812 19:56:16.556041 13191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41219 every 8 connection(s)
I20260812 19:56:16.558017 13109 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:16.564704 13192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:56:16.568329 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13109
I20260812 19:56:16.568567 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:56:16.574956 13192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:16 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
I20260812 19:56:16.602686 13192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
W20260812 19:56:16.604096 13132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
W20260812 19:56:16.606065 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:16.655475 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } attempt: 1
W20260812 19:56:16.657799 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:16.703322 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43895
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:16.726136 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } attempt: 2
W20260812 19:56:16.728438 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
I20260812 19:56:16.819775 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } attempt: 3
W20260812 19:56:16.822062 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
I20260812 19:56:16.965417 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } attempt: 4
W20260812 19:56:16.967891 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111)
W20260812 19:56:17.078579 13201 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:17.078976 13201 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:17.079067 13201 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:17.088074 13201 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:17.088207 13201 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:17.088266 13201 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:17.088315 13201 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:17.099763 13201 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:17.101655 13201 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:17.103242 13201 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:17.112489 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:17.112447 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:17.113957 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:17.114073 13201 server_base.cc:1061] running on GCE node
I20260812 19:56:17.115065 13201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:17.116740 13201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:17.117969 13201 hybrid_clock.cc:648] HybridClock initialized: now 1786564577117892 us; error 76 us; skew 500 ppm
I20260812 19:56:17.118379 13201 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:17.122131 13201 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:56:17.124089 13201 webserver.cc:533] Webserver started at http://127.25.14.61:34757/ using document root <none> and password file <none>
I20260812 19:56:17.124935 13201 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:17.125084 13201 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:17.125479 13201 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:17.128281 13201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "97dce1e214794c5cb32c247afeac6961"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.129176 13201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "97dce1e214794c5cb32c247afeac6961"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.135206 13201 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 19:56:17.139514 13222 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:17.141130 13201 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 19:56:17.141377 13201 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "97dce1e214794c5cb32c247afeac6961"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.141613 13201 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:17.160821 13201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:17.165681 13201 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:17.166078 13201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:17.190348 13201 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:43895
I20260812 19:56:17.190377 13282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:43895 every 8 connection(s)
I20260812 19:56:17.192255 13201 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:56:17.195277 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13201
I20260812 19:56:17.195539 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:56:17.198195 13283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:56:17.207926 13283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
I20260812 19:56:17.236207 13283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:17.252964 13283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:56:17.254612 13225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
W20260812 19:56:17.256481 13283 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.267246 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:17.282094 13192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:56:17.284418 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 19:56:17.305857 13283 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 1
W20260812 19:56:17.308198 13283 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:17.334558 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42033
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:17.335667 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 1
W20260812 19:56:17.338111 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.376497 13283 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 2
W20260812 19:56:17.378823 13283 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.406409 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 2
W20260812 19:56:17.408789 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.470140 13283 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 3
W20260812 19:56:17.472463 13283 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.515132 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 3
W20260812 19:56:17.517666 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.615753 13283 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 4
W20260812 19:56:17.618116 13283 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
I20260812 19:56:17.694960 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 4
W20260812 19:56:17.697190 13192 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111)
W20260812 19:56:17.712002 13298 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:17.712363 13298 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:17.712471 13298 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:17.721235 13298 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:17.721351 13298 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:17.721415 13298 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:17.721467 13298 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:17.732878 13298 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:17.734757 13298 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:17.736361 13298 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:17.745774 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:17.745910 13306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:17.746068 13304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:17.746932 13298 server_base.cc:1061] running on GCE node
I20260812 19:56:17.747746 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:17.749279 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:17.750489 13298 hybrid_clock.cc:648] HybridClock initialized: now 1786564577750436 us; error 41 us; skew 500 ppm
I20260812 19:56:17.750876 13298 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:17.754437 13298 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:56:17.756278 13298 webserver.cc:533] Webserver started at http://127.25.14.60:35021/ using document root <none> and password file <none>
I20260812 19:56:17.757068 13298 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:17.757210 13298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:17.757581 13298 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:17.760200 13298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "7590fe7dbe474deaa8d60be3a0f85d56"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.761045 13298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "7590fe7dbe474deaa8d60be3a0f85d56"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.767025 13298 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260812 19:56:17.770985 13314 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:17.772516 13298 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:17.772733 13298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "7590fe7dbe474deaa8d60be3a0f85d56"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:17.772964 13298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:17.806280 13298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:17.810437 13298 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:17.810752 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:17.833406 13298 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:42033
I20260812 19:56:17.833437 13375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:42033 every 8 connection(s)
I20260812 19:56:17.835210 13298 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:56:17.836877 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13298
I20260812 19:56:17.837090 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:56:17.841650 13376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:17.854558 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:17.881300 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:17.895146 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:17.913703 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Bootstrap starting.
I20260812 19:56:17.917186 13376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:17.917436 13283 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 5
I20260812 19:56:17.918771 13376 log.cc:826] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:17.922477 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: No bootstrap required, opened a new log
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:17.929708 13376 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:17.930078 13376 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:17.930238 13376 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7590fe7dbe474deaa8d60be3a0f85d56, State: Initialized, Role: FOLLOWER
I20260812 19:56:17.931087 13376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:17.932647 13385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:17.933092 13385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:17.934142 13376 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:17.940271 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Bootstrap starting.
I20260812 19:56:17.946015 13283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:17.947609 13283 log.cc:826] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:17.942776 13298 master_runner.cc:187] Error getting master registration for 127.25.14.62:41219: OK, instance_id { permanent_uuid: "fb726480930f458aa535cda859043cc8" instance_seqno: 1786564576549999 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:56:17.949865 13298 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:17.951342 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: No bootstrap required, opened a new log
W20260812 19:56:17.951460 13399 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:17.951591 13399 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:56:17.957576 13283 raft_consensus.cc:359] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:17.957958 13283 raft_consensus.cc:385] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:17.958125 13283 raft_consensus.cc:740] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dce1e214794c5cb32c247afeac6961, State: Initialized, Role: FOLLOWER
I20260812 19:56:17.958792 13283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:17.959982 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:17.960331 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:17.961261 13283 sys_catalog.cc:565] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:56:17.973291 13416 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:17.973466 13416 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:56:17.974452 13192 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } attempt: 5
I20260812 19:56:17.969661 13201 master_runner.cc:187] Error getting master registration for 127.25.14.62:41219: OK, instance_id { permanent_uuid: "fb726480930f458aa535cda859043cc8" instance_seqno: 1786564576549999 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:56:17.982018 13201 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 19:56:17 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:18.000541 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap starting.
I20260812 19:56:18.005295 13192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:18.006947 13192 log.cc:826] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:18.013300 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: No bootstrap required, opened a new log
I20260812 19:56:18.021976 13192 raft_consensus.cc:359] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:18.022385 13192 raft_consensus.cc:385] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:18.022534 13192 raft_consensus.cc:740] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb726480930f458aa535cda859043cc8, State: Initialized, Role: FOLLOWER
I20260812 19:56:18.023366 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:18.024580 13419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.024982 13419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:18.026063 13192 sys_catalog.cc:565] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:56:18.036906 13138 tablet.cc:2406] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:56:18.049232 13109 master_runner.cc:264] No leader master found from master fb726480930f458aa535cda859043cc8
I20260812 19:56:18.049407 13109 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:56:18.055475 13432 catalog_manager.cc:1594] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:18.055645 13432 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:56:18.063154 13321 tablet.cc:2406] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:56:18.153365 13385 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:56:18.153765 13385 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:18.155821 13385 leader_election.cc:290] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb726480930f458aa535cda859043cc8 (127.25.14.62:41219), 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895)
I20260812 19:56:18.158417 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961" is_pre_election: true
I20260812 19:56:18.158987 13253 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 0.
I20260812 19:56:18.159996 13318 leader_election.cc:304] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7590fe7dbe474deaa8d60be3a0f85d56, 97dce1e214794c5cb32c247afeac6961; no voters:
I20260812 19:56:18.157407 13161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb726480930f458aa535cda859043cc8" is_pre_election: true
I20260812 19:56:18.160986 13161 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 0.
I20260812 19:56:18.162212 13385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:56:18.162396 13385 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:56:18.162503 13385 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:18.166695 13385 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
W20260812 19:56:18.169229 13229 tablet.cc:2406] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:56:18.173283 13161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb726480930f458aa535cda859043cc8"
I20260812 19:56:18.173547 13161 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:18.177395 13161 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 1.
I20260812 19:56:18.178179 13318 leader_election.cc:304] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7590fe7dbe474deaa8d60be3a0f85d56, fb726480930f458aa535cda859043cc8; no voters:
I20260812 19:56:18.181229 13385 leader_election.cc:290] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 1 election: Requested vote from peers fb726480930f458aa535cda859043cc8 (127.25.14.62:41219), 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895)
I20260812 19:56:18.182000 13385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:56:18.182528 13385 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 LEADER]: Becoming Leader. State: Replica: 7590fe7dbe474deaa8d60be3a0f85d56, State: Running, Role: LEADER
I20260812 19:56:18.183398 13385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:18.207086 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961"
I20260812 19:56:18.207389 13253 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:18.212067 13253 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 1.
I20260812 19:56:18.227043 13434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.227715 13434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:18.233562 13439 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:18.238400 13439 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:18.260144 13253 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Refusing update from remote peer 7590fe7dbe474deaa8d60be3a0f85d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:56:18.261492 13434 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:56:18.269093 13161 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Refusing update from remote peer 7590fe7dbe474deaa8d60be3a0f85d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:56:18.270546 13434 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:56:18.311400 13419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.311738 13419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:18.318614 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.318964 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:18.319187 13440 mvcc.cc:204] Tried to move back new op lower bound from 7317768512538165248 to 7317768512254038016. Current Snapshot: MvccSnapshot[applied={T|T < 7317768512538165248}]
I20260812 19:56:18.338178 13419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.338449 13419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:18.339003 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.339243 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:18.343879 13385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.344206 13385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:18.344515 13385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:18.344733 13385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:18.346573 13439 catalog_manager.cc:1383] Generated new cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:18.346688 13439 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:18.394184 13439 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:56:18.397320 13439 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:18.425393 13439 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Generated new TSK 0
I20260812 19:56:18.426934 13439 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 19:56:18.648665 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.115s user 0.000s sys 0.024s
I20260812 19:56:18.649870 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:18.975622 13416 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:18.975732 13416 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: loading cluster ID for follower catalog manager: success
I20260812 19:56:18.978771 13416 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: acquiring CA information for follower catalog manager: success
I20260812 19:56:18.981097 13416 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:56:19.046478 13458 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:19.046926 13458 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:19.047106 13458 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:19.056007 13458 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:19.056295 13458 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:19.057837 13432 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:19.057922 13432 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: loading cluster ID for follower catalog manager: success
I20260812 19:56:19.060767 13432 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: acquiring CA information for follower catalog manager: success
I20260812 19:56:19.062947 13432 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:19.068230 13458 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:19.070006 13458 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:19.071480 13458 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:19.082470 13464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:19.082679 13465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:19.083288 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:19.083640 13458 server_base.cc:1061] running on GCE node
I20260812 19:56:19.084273 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:19.085358 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:19.086581 13458 hybrid_clock.cc:648] HybridClock initialized: now 1786564579086512 us; error 59 us; skew 500 ppm
I20260812 19:56:19.086959 13458 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:19.090559 13458 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:19.092474 13458 webserver.cc:533] Webserver started at http://127.25.14.1:43121/ using document root <none> and password file <none>
I20260812 19:56:19.093294 13458 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:19.093451 13458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:19.093873 13458 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:19.096449 13458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.097317 13458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.103330 13458 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 19:56:19.107043 13475 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:19.108477 13458 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:56:19.108677 13458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.108893 13458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:19.137840 13458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:19.142263 13458 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:19.142588 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:19.143803 13458 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:19.145545 13458 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:19.145673 13458 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 19:56:19.145828 13458 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:19.145905 13458 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:19.183084 13458 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43851
I20260812 19:56:19.183128 13611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43851 every 8 connection(s)
I20260812 19:56:19.184862 13458 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:19.194779 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13458
I20260812 19:56:19.195034 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:19.219871 13615 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:19.220327 13615 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.221526 13615 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:19.224241 13613 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:19.224380 13613 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.224634 13241 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:19.224937 13613 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:19.226945 13149 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:19.232180 13612 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:19.232378 13612 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.232841 13612 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:19.234953 13333 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:19.237529 13333 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:50151
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:19.346405 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:19.733045 13629 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:19.733407 13629 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:19.733500 13629 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:19.742344 13629 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:19.742558 13629 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:19.754185 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:19.755927 13629 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:19.757372 13629 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:19.769649 13636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:19.769652 13637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:19.770282 13639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:19.771554 13629 server_base.cc:1061] running on GCE node
I20260812 19:56:19.772289 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:19.773458 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:19.774734 13629 hybrid_clock.cc:648] HybridClock initialized: now 1786564579774682 us; error 65 us; skew 500 ppm
I20260812 19:56:19.775151 13629 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:19.779178 13629 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:19.781363 13629 webserver.cc:533] Webserver started at http://127.25.14.2:37455/ using document root <none> and password file <none>
I20260812 19:56:19.782353 13629 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:19.782539 13629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:19.782955 13629 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:19.786032 13629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "1dab731b68f1479fa19aa02c810c1a16"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.787067 13629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "1dab731b68f1479fa19aa02c810c1a16"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.794202 13629 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 19:56:19.798475 13648 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:19.800551 13629 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:19.800796 13629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1dab731b68f1479fa19aa02c810c1a16"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:19.801041 13629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:19.835306 13629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:19.840135 13629 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:19.840466 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:19.841555 13629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:19.843564 13629 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:19.843698 13629 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:19.843873 13629 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:19.843961 13629 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:19.882263 13629 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:38889
I20260812 19:56:19.882308 13785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:38889 every 8 connection(s)
I20260812 19:56:19.883852 13629 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:19.890317 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13629
I20260812 19:56:19.890618 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:19 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:19.932104 13787 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:19.932169 13786 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:19.932595 13787 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.932600 13786 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.934068 13786 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:19.934100 13787 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:19.936185 13333 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:38889)
I20260812 19:56:19.936635 13149 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:38889)
I20260812 19:56:19.937469 13333 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:33603
I20260812 19:56:19.942147 13788 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:19.942328 13788 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:19.942745 13788 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:19.944037 13241 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:38889)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:20.054620 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:20.240741 13612 heartbeater.cc:499] Master 127.25.14.60:42033 was elected leader, sending a full tablet report...
W20260812 19:56:20.447212 13803 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:20.447604 13803 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:20.447793 13803 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:20.457087 13803 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:20.457294 13803 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:20.469338 13803 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:20.471238 13803 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:20.472697 13803 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:20.483497 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:20.483457 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:20.484329 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:20.485545 13803 server_base.cc:1061] running on GCE node
I20260812 19:56:20.486243 13803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:20.487390 13803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:20.488626 13803 hybrid_clock.cc:648] HybridClock initialized: now 1786564580488586 us; error 37 us; skew 500 ppm
I20260812 19:56:20.489050 13803 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:20 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:20.492833 13803 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:20.494671 13803 webserver.cc:533] Webserver started at http://127.25.14.3:46049/ using document root <none> and password file <none>
I20260812 19:56:20.495463 13803 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:20.495563 13803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:20.495867 13803 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:20.498584 13803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "8f28d67541994e998f6d8955c79f1144"
format_stamp: "Formatted at 2026-08-12 19:56:20 on dist-test-slave-p3t5"
I20260812 19:56:20.499450 13803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "8f28d67541994e998f6d8955c79f1144"
format_stamp: "Formatted at 2026-08-12 19:56:20 on dist-test-slave-p3t5"
I20260812 19:56:20.505542 13803 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:56:20.509493 13819 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:20.511222 13803 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 19:56:20.511441 13803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "8f28d67541994e998f6d8955c79f1144"
format_stamp: "Formatted at 2026-08-12 19:56:20 on dist-test-slave-p3t5"
I20260812 19:56:20.511652 13803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:20.530889 13803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:20.535499 13803 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:20.535830 13803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:20.536980 13803 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:20.539085 13803 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:20.539170 13803 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:20.539315 13803 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:20.539368 13803 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:20.583755 13803 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:42801
I20260812 19:56:20.583807 13958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:42801 every 8 connection(s)
I20260812 19:56:20.585484 13803 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:20.590930 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13803
I20260812 19:56:20.591189 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:20 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:20 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:20 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:20.615661 13962 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:20.616099 13962 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:20.617386 13962 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:20.620035 13240 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:42801)
I20260812 19:56:20.621104 13959 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:20.621282 13959 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:20.621948 13959 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:20.623099 13333 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:42801)
I20260812 19:56:20.623227 13961 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:20.623400 13961 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:20.623837 13961 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:20.624138 13333 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:51021
I20260812 19:56:20.624979 13149 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:42801)
I20260812 19:56:20.637529 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:20.646178 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:20.646445 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13109
I20260812 19:56:20.647473 13186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:20.762012 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13109
I20260812 19:56:20.794411 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--webserver_interface=127.25.14.62
--webserver_port=46841
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:20.940335 13787 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111)
I20260812 19:56:20.940697 13786 heartbeater.cc:499] Master 127.25.14.60:42033 was elected leader, sending a full tablet report...
W20260812 19:56:21.179484 13972 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:21.179891 13972 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:21.180063 13972 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:21.191148 13972 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:21.191298 13972 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:21.191377 13972 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:21.191452 13972 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:21.207381 13972 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=46841
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:21.209124 13972 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:21.210673 13972 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:21.220157 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:21.220263 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:21.220160 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:21.221365 13972 server_base.cc:1061] running on GCE node
I20260812 19:56:21.222241 13972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:21.223717 13972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:21.224929 13972 hybrid_clock.cc:648] HybridClock initialized: now 1786564581224888 us; error 32 us; skew 500 ppm
I20260812 19:56:21.225324 13972 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:21 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:21.229094 13972 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:21.231422 13972 webserver.cc:533] Webserver started at http://127.25.14.62:46841/ using document root <none> and password file <none>
W20260812 19:56:21.231613 13613 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111)
I20260812 19:56:21.232504 13972 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:21.232709 13972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:21.238919 13972 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:21.242295 13993 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:21.244050 13972 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:56:21.244279 13972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:21.245347 13972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 19:56:21.251050 13318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111)
W20260812 19:56:21.252830 13318 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 -> Peer fb726480930f458aa535cda859043cc8 (127.25.14.62:41219): Couldn't send request to peer fb726480930f458aa535cda859043cc8. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:56:21.266151 13972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:21.270316 13972 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:21.270640 13972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:21.294217 13972 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41219
I20260812 19:56:21.294229 14055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41219 every 8 connection(s)
I20260812 19:56:21.296098 13972 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:21.296463 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13972
I20260812 19:56:21.296806 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13458
I20260812 19:56:21.304924 14057 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:56:21 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:21.309193 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap starting.
I20260812 19:56:21.316771 13607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:21.333792 14057 log.cc:826] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:21.346462 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:21.347293 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap complete.
I20260812 19:56:21.358486 14057 raft_consensus.cc:359] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:21.359606 14057 raft_consensus.cc:740] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb726480930f458aa535cda859043cc8, State: Initialized, Role: FOLLOWER
I20260812 19:56:21.360668 14057 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:21.361819 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:21.362195 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:21.363336 14057 sys_catalog.cc:565] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:21.375576 14078 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:21.375713 14078 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: loading cluster ID for follower catalog manager: success
I20260812 19:56:21.380757 14078 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: acquiring CA information for follower catalog manager: success
I20260812 19:56:21.384126 14078 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:21.436506 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13458
I20260812 19:56:21.462113 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43851
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43121
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:21.626684 13959 heartbeater.cc:499] Master 127.25.14.60:42033 was elected leader, sending a full tablet report...
I20260812 19:56:21.638581 14016 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8f28d67541994e998f6d8955c79f1144" instance_seqno: 1786564580574284) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:45109; Asking this server to re-register.
I20260812 19:56:21.639663 13961 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:21.641763 14016 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:42801)
I20260812 19:56:21.727710 13459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:21.728024 13459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:21.742590 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:21.742916 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:21.903795 14084 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:21.904227 14084 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:21.904392 14084 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:21.913501 14084 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:21.913797 14084 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:21.925580 14084 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43121
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:21.927482 14084 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:21.929250 14084 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:21.940336 14094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:21.940366 14093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:21.941550 14096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:21.942085 14084 server_base.cc:1061] running on GCE node
I20260812 19:56:21.942808 14084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:21.944104 14084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:21.945554 14084 hybrid_clock.cc:648] HybridClock initialized: now 1786564581945480 us; error 67 us; skew 500 ppm
I20260812 19:56:21.946023 14084 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:21 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:21.950997 14084 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:21.953200 13787 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:21.954188 14084 webserver.cc:533] Webserver started at http://127.25.14.1:43121/ using document root <none> and password file <none>
I20260812 19:56:21.954423 14016 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1dab731b68f1479fa19aa02c810c1a16" instance_seqno: 1786564579873021) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:58335; Asking this server to re-register.
I20260812 19:56:21.954988 13787 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:21.955147 14084 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:21.955308 14084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:21.956194 14016 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:38889)
I20260812 19:56:21.961536 14084 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:21.964618 14105 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:21.966260 14084 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:21.966466 14084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:21.967270 14084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:22.007258 14084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:22.011933 14084 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:22.012279 14084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:22.013444 14084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:22.015264 14084 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:22.015409 14084 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:22.015560 14084 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:22.015651 14084 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:22.053948 14084 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43851
I20260812 19:56:22.054041 14239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43851 every 8 connection(s)
I20260812 19:56:22.055676 14084 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:22.062291 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14084
Aug 12 19:56:22 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:22 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:22 dist-test-slave-p3t5 krb5kdc[13090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:22.099134 14240 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:22.099573 14240 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:22.100538 14240 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:22.101459 14241 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:22.101658 14241 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:22.102262 14241 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:22.103854 14016 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:22.103919 13333 ts_manager.cc:194] Re-registered known tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:22.105062 13333 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:51825
I20260812 19:56:22.106284 14242 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:22.106458 14242 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:22.107077 14242 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:22.108536 13240 ts_manager.cc:194] Re-registered known tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:43851)
I20260812 19:56:22.110260 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:22.110574 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:22.132140 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14084
W20260812 19:56:22.133328 14254 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:22.129776 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.25.14.1:43851)
0812 19:56:22.130499 (+ 723us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:22.130506 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:22.130537 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:56:22.130612 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:22.130619 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:22.130835 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:22.131021 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:22.131961 (+ 940us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55590: BlockingRecv error: recv got EOF from 127.0.0.1:55590 (error 108)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":382,"threads_started":1}
W20260812 19:56:22.134685 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14084: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:43851: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:22.134779 14254 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:22.133061 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.25.14.1:43851)
0812 19:56:22.133710 (+ 649us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:22.133715 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:22.133733 (+ 18us) server_negotiation.cc:408] Connection header received
0812 19:56:22.133788 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:22.133794 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:22.133865 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:22.133959 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:22.134541 (+ 582us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55596: BlockingRecv error: recv got EOF from 127.0.0.1:55596 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":405,"threads_started":1}
I20260812 19:56:22.134887 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14084
I20260812 19:56:22.161137 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43851
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43121
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:22.541123 14257 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:22.541507 14257 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:22.541678 14257 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:22.541764 14257 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:22.550822 14257 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:22.551131 14257 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:22.566331 14257 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43121
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:22.568308 14257 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:22.569962 14257 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:22.581167 14264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:22.581372 14262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:22.581712 14267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:22.582451 14257 server_base.cc:1061] running on GCE node
I20260812 19:56:22.583105 14257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:22.584275 14257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:22.585552 14257 hybrid_clock.cc:648] HybridClock initialized: now 1786564582585464 us; error 62 us; skew 500 ppm
I20260812 19:56:22.585940 14257 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:22 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564582, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:22.589807 14257 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:22.592298 14257 webserver.cc:533] Webserver started at http://127.25.14.1:43121/ using document root <none> and password file <none>
I20260812 19:56:22.593235 14257 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:22.593335 14257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:22.599957 14257 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:56:22.603487 14274 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:22.605360 14257 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:22.605584 14257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:22.606467 14257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:22.637996 14257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:22.642409 14257 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:22.642736 14257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:22.643953 14257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:22.645893 14257 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:22.645972 14257 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:22.646152 14257 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:22.646204 14257 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:22.646692 14257 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:22.646875 14257 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:22.646979 14257 fault_injection.cc:59] injecting fault for kudu (pid 14257): FLAGS_fault_before_start (process will exit)
I20260812 19:56:22.674603 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:43851
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43121
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:23.064853 14282 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:23.065332 14282 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:23.065501 14282 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:23.074903 14282 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:23.075191 14282 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:23.086570 14282 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43121
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:23.088411 14282 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:23.089915 14282 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:23.100507 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:23.101094 14295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:23.100512 14293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:23.101962 14282 server_base.cc:1061] running on GCE node
I20260812 19:56:23.102665 14282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:23.103753 14282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:23.105019 14282 hybrid_clock.cc:648] HybridClock initialized: now 1786564583104969 us; error 47 us; skew 500 ppm
I20260812 19:56:23.105381 14282 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:23 dist-test-slave-p3t5 krb5kdc[13090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564583, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:23.109193 14282 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:23.111016 14282 webserver.cc:533] Webserver started at http://127.25.14.1:43121/ using document root <none> and password file <none>
I20260812 19:56:23.111791 14282 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:23.111902 14282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:23.118073 14282 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:23.121102 14303 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:23.122817 14282 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:23.122977 14282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:23.123744 14282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:23.145239 14282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:23.149667 14282 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:23.149928 14282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:23.151141 14282 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:23.152858 14282 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:23.152951 14282 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:23.153033 14282 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:23.153090 14282 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:23.191112 14282 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:43851
I20260812 19:56:23.191164 14440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:43851 every 8 connection(s)
I20260812 19:56:23.192788 14282 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:23.198118 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14282
I20260812 19:56:23.198596 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14282
W20260812 19:56:23.213987 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14282: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:43851: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:23.214190 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14282
I20260812 19:56:23.238489 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13629
W20260812 19:56:23.239248 14285 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:23.196893 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:38267 (local address 127.25.14.62:41219)
0812 19:56:23.197106 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:23.197113 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:23.209274 (+ 12161us) server_negotiation.cc:408] Connection header received
0812 19:56:23.211950 (+ 2676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:23.211960 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:23.212055 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:23.212155 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:23.238222 (+ 26067us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:38267: BlockingRecv error: recv got EOF from 127.25.14.1:38267 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 19:56:23.239275 14445 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:23.196663 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48893 (local address 127.25.14.60:42033)
0812 19:56:23.206590 (+ 9927us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:23.206600 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:23.209892 (+ 3292us) server_negotiation.cc:408] Connection header received
0812 19:56:23.211901 (+ 2009us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:23.211912 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:23.212034 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:23.212172 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:23.238247 (+ 26075us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48893: BlockingRecv error: recv got EOF from 127.25.14.1:48893 (error 108)
Metrics: {"server-negotiator.queue_time_us":9761,"thread_start_us":9631,"threads_started":1}
W20260812 19:56:23.239300 14452 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:23.211019 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48001 (local address 127.25.14.61:43895)
0812 19:56:23.216849 (+ 5830us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:23.216859 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:23.238253 (+ 21394us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48001: BlockingRecv error: recv got EOF from 127.25.14.1:48001 (error 108)
Metrics: {"server-negotiator.queue_time_us":5637,"thread_start_us":6255,"threads_started":1}
W20260812 19:56:23.241936 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13629: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:38889: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:23.242116 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13629
I20260812 19:56:23.267067 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13803
W20260812 19:56:23.270161 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13803: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:42801: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:23.270380 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13803
I20260812 19:56:23.293104 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13972
W20260812 19:56:23.295538 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13972: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:41219: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:23.295667 14285 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:23.294208 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.25.14.62:41219)
0812 19:56:23.294414 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:23.294420 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:23.294443 (+ 23us) server_negotiation.cc:408] Connection header received
0812 19:56:23.294520 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:23.294529 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:23.294622 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:23.294739 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:23.295440 (+ 701us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60672: BlockingRecv error: recv got EOF from 127.0.0.1:60672 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 19:56:23.295701 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13972
I20260812 19:56:23.325225 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13201
I20260812 19:56:23.326771 13277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:23.441130 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13201
W20260812 19:56:23.449478 13318 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 -> Peer fb726480930f458aa535cda859043cc8 (127.25.14.62:41219): Couldn't send request to peer fb726480930f458aa535cda859043cc8. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:56:23.470917 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13298
I20260812 19:56:23.472206 13371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:23.588436 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13298
2026-08-12T19:56:23Z chronyd exiting
Aug 12 19:56:23 dist-test-slave-p3t5 krb5kdc[14460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:23 dist-test-slave-p3t5 krb5kdc[14460](info): set up 2 sockets
Aug 12 19:56:23 dist-test-slave-p3t5 krb5kdc[14460](info): commencing operation
krb5kdc: starting...
W20260812 19:56:25.730885 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.098s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:25 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:56:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:26.078616 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:26.461549 14477 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:26.462037 14477 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:26.462137 14477 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:26.470954 14477 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:26.471071 14477 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:26.471125 14477 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:26.471184 14477 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:26.483016 14477 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:26.484853 14477 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:26.486330 14477 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:26.495555 14482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:26.495599 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:26.496331 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:26.496737 14477 server_base.cc:1061] running on GCE node
I20260812 19:56:26.497557 14477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:26.499142 14477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:26.500404 14477 hybrid_clock.cc:648] HybridClock initialized: now 1786564586500352 us; error 44 us; skew 500 ppm
I20260812 19:56:26.500828 14477 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:26 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564586, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:26.504772 14477 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:26.506762 14477 webserver.cc:533] Webserver started at http://127.25.14.62:45589/ using document root <none> and password file <none>
I20260812 19:56:26.507591 14477 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:26.507731 14477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:26.514103 14477 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:26.517584 14494 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:26.519244 14477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:26.519485 14477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:26.520309 14477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:26.538951 14477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:26.543538 14477 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:26.543843 14477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:26.571671 14477 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41219
I20260812 19:56:26.571725 14562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41219 every 8 connection(s)
I20260812 19:56:26.573516 14477 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:26.582326 14563 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:26.583539 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14477
I20260812 19:56:26.583766 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:56:26.586639 14563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap starting.
I20260812 19:56:26.606994 14563 log.cc:826] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:26.617985 14563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:26.618664 14563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap complete.
I20260812 19:56:26.627364 14563 raft_consensus.cc:359] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:26.628486 14563 raft_consensus.cc:740] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb726480930f458aa535cda859043cc8, State: Initialized, Role: FOLLOWER
I20260812 19:56:26.629419 14563 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:26.630563 14570 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:26.630906 14570 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:26.631978 14563 sys_catalog.cc:565] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:26.644832 14586 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:26.645072 14586 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: loading cluster ID for follower catalog manager: success
I20260812 19:56:26.650595 14586 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: acquiring CA information for follower catalog manager: success
I20260812 19:56:26.654412 14586 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:26.673785 14477 master_runner.cc:187] Error getting master registration for 127.25.14.61:43895: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43895: connect: Connection refused (error 111),
I20260812 19:56:26.680460 14477 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:26.720974 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43895
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:27.105103 14593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:27.105508 14593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:27.105697 14593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:27.114722 14593 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:27.114852 14593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:27.114952 14593 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:27.115054 14593 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:27.126899 14593 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:27.128722 14593 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:27.130277 14593 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:27.139657 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:27.139688 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:27.139969 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:27.143177 14593 server_base.cc:1061] running on GCE node
I20260812 19:56:27.144106 14593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:27.145722 14593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:27.146960 14593 hybrid_clock.cc:648] HybridClock initialized: now 1786564587146887 us; error 65 us; skew 500 ppm
I20260812 19:56:27.147346 14593 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:27.151175 14593 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:56:27.152899 14593 webserver.cc:533] Webserver started at http://127.25.14.61:43491/ using document root <none> and password file <none>
I20260812 19:56:27.153770 14593 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:27.153899 14593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:27.159861 14593 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:56:27.163116 14611 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:27.164655 14593 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:56:27.164829 14593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "97dce1e214794c5cb32c247afeac6961"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:27.165812 14593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:27.191769 14593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:27.201597 14593 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:27.201989 14593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:27.225493 14593 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:43895
I20260812 19:56:27.225512 14676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:43895 every 8 connection(s)
I20260812 19:56:27.227532 14593 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:56:27.236183 14678 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:27.236840 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14593
I20260812 19:56:27.237092 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:56:27.241959 14678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Bootstrap starting.
I20260812 19:56:27.263093 14678 log.cc:826] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:27.275672 14678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:27.276449 14678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: Bootstrap complete.
I20260812 19:56:27.284611 14678 raft_consensus.cc:359] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:27.285462 14678 raft_consensus.cc:740] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dce1e214794c5cb32c247afeac6961, State: Initialized, Role: FOLLOWER
I20260812 19:56:27.286417 14678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:27.287537 14683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:27.288015 14683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:27.288827 14678 sys_catalog.cc:565] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:27.301192 14696 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:27.301491 14696 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: loading cluster ID for follower catalog manager: success
I20260812 19:56:27.307648 14696 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: acquiring CA information for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:27.311349 14696 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:27.335134 14593 master_runner.cc:187] Error getting master registration for 127.25.14.60:42033: Network error: Client connection negotiation failed: client connection to 127.25.14.60:42033: connect: Connection refused (error 111),
I20260812 19:56:27.339674 14593 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:27.389935 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42033
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:27.791774 14703 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:27.792207 14703 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:27.792366 14703 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:27.801286 14703 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:27.801415 14703 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:27.801515 14703 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:27.801595 14703 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:27.813138 14703 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:27.815102 14703 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:27.816664 14703 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:27.825791 14709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:27.825734 14710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:27.826274 14714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:27.827108 14703 server_base.cc:1061] running on GCE node
I20260812 19:56:27.827991 14703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:27.829516 14703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:27.830798 14703 hybrid_clock.cc:648] HybridClock initialized: now 1786564587830718 us; error 63 us; skew 500 ppm
I20260812 19:56:27.831230 14703 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:27.835084 14703 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:56:27.837174 14703 webserver.cc:533] Webserver started at http://127.25.14.60:37799/ using document root <none> and password file <none>
I20260812 19:56:27.838160 14703 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:27.838296 14703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:27.844718 14703 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 19:56:27.848120 14722 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:27.849897 14703 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:27.850121 14703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "7590fe7dbe474deaa8d60be3a0f85d56"
format_stamp: "Formatted at 2026-08-12 19:56:17 on dist-test-slave-p3t5"
I20260812 19:56:27.850940 14703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:27.868140 14703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:27.872781 14703 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:27.873100 14703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:27.895886 14703 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:42033
I20260812 19:56:27.895905 14781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:42033 every 8 connection(s)
I20260812 19:56:27.898048 14703 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:56:27.899024 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14703
I20260812 19:56:27.899260 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:56:27.909518 14782 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:27.917816 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Bootstrap starting.
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:27.939577 14782 log.cc:826] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:27.950980 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:27.951622 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Bootstrap complete.
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:27.959785 14782 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:27.960629 14782 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7590fe7dbe474deaa8d60be3a0f85d56, State: Initialized, Role: FOLLOWER
I20260812 19:56:27.961393 14782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:27.962451 14789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:27.962786 14789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:27.963821 14782 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:27.976317 14804 raft_consensus.cc:493] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:56:27.977447 14804 raft_consensus.cc:515] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:27.980399 14801 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:27.980576 14801 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: loading cluster ID for follower catalog manager: success
I20260812 19:56:27.984123 14804 leader_election.cc:290] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895), 7590fe7dbe474deaa8d60be3a0f85d56 (127.25.14.60:42033)
Aug 12 19:56:27 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:27.996384 14801 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: acquiring CA information for follower catalog manager: success
I20260812 19:56:28.005213 14801 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:28.010186 14642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb726480930f458aa535cda859043cc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961" is_pre_election: true
I20260812 19:56:28.010927 14642 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb726480930f458aa535cda859043cc8 in term 1.
I20260812 19:56:28.012071 14500 leader_election.cc:304] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97dce1e214794c5cb32c247afeac6961, fb726480930f458aa535cda859043cc8; no voters:
I20260812 19:56:28.012732 14804 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:56:28.012952 14804 raft_consensus.cc:493] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:56:28.013103 14804 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:28.019737 14754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb726480930f458aa535cda859043cc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" is_pre_election: true
I20260812 19:56:28.020473 14754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb726480930f458aa535cda859043cc8 in term 1.
I20260812 19:56:28.020985 14804 raft_consensus.cc:515] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:28.023639 14804 leader_election.cc:290] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [CANDIDATE]: Term 2 election: Requested vote from peers 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895), 7590fe7dbe474deaa8d60be3a0f85d56 (127.25.14.60:42033)
I20260812 19:56:28.024467 14642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb726480930f458aa535cda859043cc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961"
I20260812 19:56:28.024735 14642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:28.025008 14754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb726480930f458aa535cda859043cc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7590fe7dbe474deaa8d60be3a0f85d56"
I20260812 19:56:28.025225 14754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:28.030719 14642 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb726480930f458aa535cda859043cc8 in term 2.
I20260812 19:56:28.031071 14754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb726480930f458aa535cda859043cc8 in term 2.
I20260812 19:56:28.032915 14498 leader_election.cc:304] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7590fe7dbe474deaa8d60be3a0f85d56, fb726480930f458aa535cda859043cc8; no voters:
I20260812 19:56:28.033537 14804 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:56:28.034276 14804 raft_consensus.cc:697] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 LEADER]: Becoming Leader. State: Replica: fb726480930f458aa535cda859043cc8, State: Running, Role: LEADER
I20260812 19:56:28.035024 14804 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:28.035537 14703 master_runner.cc:233] Existing masters have differing terms: 1 vs 2
I20260812 19:56:28.035815 14703 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:28.038949 14808 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb726480930f458aa535cda859043cc8. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.039224 14808 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:28.040099 14810 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:28.042394 14810 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:28.043323 14810 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:28.043412 14810 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:28.044906 14810 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:28.045909 14810 catalog_manager.cc:6103] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Loaded TSK: 0
I20260812 19:56:28.046967 14810 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:28.197220 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:28.511958 14754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Refusing update from remote peer fb726480930f458aa535cda859043cc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:28.513200 14808 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:28.517601 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb726480930f458aa535cda859043cc8. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.517958 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:28.520918 14642 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 2 FOLLOWER]: Refusing update from remote peer fb726480930f458aa535cda859043cc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:28.522153 14804 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.522446 14804 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:28.522650 14804 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:28.531774 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.532043 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:28.539721 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.540199 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:28.540084 14828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb726480930f458aa535cda859043cc8. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:28.540431 14828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:28.603113 14817 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:28.603549 14817 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:28.603718 14817 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:28.612497 14817 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:28.612799 14817 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:28.624044 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:28.626134 14817 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:28.627784 14817 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:28.638533 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:28.638577 14838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:28.639580 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:28.639991 14817 server_base.cc:1061] running on GCE node
I20260812 19:56:28.640645 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:28.641726 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:28.643011 14817 hybrid_clock.cc:648] HybridClock initialized: now 1786564588642952 us; error 44 us; skew 500 ppm
I20260812 19:56:28.643400 14817 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:28 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564588, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:28.647492 14817 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:28.649456 14817 webserver.cc:533] Webserver started at http://127.25.14.1:43141/ using document root <none> and password file <none>
I20260812 19:56:28.650414 14817 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:28.650581 14817 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:28.656775 14817 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 19:56:28.659936 14848 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:28.661718 14817 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 19:56:28.661927 14817 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:28.663130 14817 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:28.680163 14817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:28.684854 14817 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:28.685115 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:28.686228 14817 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:28.687927 14817 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:28.688037 14817 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:28.688134 14817 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:28.688190 14817 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:28.726065 14817 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40223
I20260812 19:56:28.726126 14978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40223 every 8 connection(s)
I20260812 19:56:28.727766 14817 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:28.731472 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14817
I20260812 19:56:28.731716 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:56:28 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564588, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:28 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564588, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:28 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564588, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:28.782092 14979 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:28.781920 14982 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:28.782648 14982 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:28.782655 14979 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:28.783650 14982 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:28.783723 14979 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:28.784986 14981 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:28.785140 14981 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:28.785755 14981 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:28.786675 14631 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
I20260812 19:56:28.786690 14742 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
I20260812 19:56:28.787629 14516 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
I20260812 19:56:28.789539 14516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:45215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:28.887427 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:29.001909 14821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fb726480930f458aa535cda859043cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:29.002296 14821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:29.274144 14998 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:29.274477 14998 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:29.274587 14998 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:29.283567 14998 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:29.283771 14998 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:29.294982 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:29.296651 14998 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:29.298112 14998 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:29.308961 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:29.308930 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:29.309660 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:29.310357 14998 server_base.cc:1061] running on GCE node
I20260812 19:56:29.311050 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:29.312080 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:29.313297 14998 hybrid_clock.cc:648] HybridClock initialized: now 1786564589313246 us; error 35 us; skew 500 ppm
I20260812 19:56:29.313724 14998 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:29 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:29.317564 14998 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:29.319543 14998 webserver.cc:533] Webserver started at http://127.25.14.2:41159/ using document root <none> and password file <none>
I20260812 19:56:29.320381 14998 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:29.320534 14998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:29.326638 14998 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:29.329734 15016 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:29.331221 14998 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:29.331410 14998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1dab731b68f1479fa19aa02c810c1a16"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:29.332173 14998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:29.354722 14998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:29.359234 14998 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:29.359546 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:29.360617 14998 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:29.362350 14998 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:29.362442 14998 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:29.362607 14998 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:29.362692 14998 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:29.399596 14998 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:41377
I20260812 19:56:29.399655 15144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:41377 every 8 connection(s)
I20260812 19:56:29.401294 14998 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:29.405144 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14998
I20260812 19:56:29.405390 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:29 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:29 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:29.440339 15150 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:29.440811 15150 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:29.442010 15146 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:29.442036 15150 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:29.442183 15146 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:29.442816 15146 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:29.444360 14742 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:41377)
I20260812 19:56:29.444705 14631 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:41377)
Aug 12 19:56:29 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:29.456902 15147 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:29.457098 15147 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:29.457715 15147 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:29.458956 14516 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:41377)
I20260812 19:56:29.459823 14516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:35719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:29.554039 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:29.792896 14981 heartbeater.cc:499] Master 127.25.14.62:41219 was elected leader, sending a full tablet report...
W20260812 19:56:29.942363 15165 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:29.942734 15165 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:29.942924 15165 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:29.951969 15165 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:29.952241 15165 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:29.963840 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:29.965775 15165 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:29.967326 15165 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:29.978852 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:29.978914 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:29.979112 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:29.980405 15165 server_base.cc:1061] running on GCE node
I20260812 19:56:29.981135 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:29.982366 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:29.983651 15165 hybrid_clock.cc:648] HybridClock initialized: now 1786564589983603 us; error 47 us; skew 500 ppm
I20260812 19:56:29.984062 15165 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:29 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:29.988291 15165 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:29.990327 15165 webserver.cc:533] Webserver started at http://127.25.14.3:34073/ using document root <none> and password file <none>
I20260812 19:56:29.991098 15165 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:29.991215 15165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:29.997416 15165 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:30.000885 15183 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:30.002636 15165 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:30.002810 15165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "8f28d67541994e998f6d8955c79f1144"
format_stamp: "Formatted at 2026-08-12 19:56:20 on dist-test-slave-p3t5"
I20260812 19:56:30.003537 15165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:30.030298 15165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:30.035045 15165 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:30.035372 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:30.036612 15165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:30.038491 15165 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:30.038745 15165 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:30.038920 15165 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:30.039027 15165 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:30.075899 15165 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:40281
I20260812 19:56:30.075932 15311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:40281 every 8 connection(s)
I20260812 19:56:30.077600 15165 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:30.078698 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15165
I20260812 19:56:30.078982 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:30.124225 15312 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:30.124658 15312 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:30.125509 15313 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:30.125712 15313 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:30.125967 15312 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:30.126302 15313 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:30.127301 15315 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:30.127450 15315 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:30.127971 15315 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:30.128628 14516 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:40281)
I20260812 19:56:30.129581 14742 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:40281)
I20260812 19:56:30.129849 14516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:35991
I20260812 19:56:30.130981 14631 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:40281)
I20260812 19:56:30.145815 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:30.153928 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:30.154141 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14477
I20260812 19:56:30.155072 14556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:30.273130 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14477
I20260812 19:56:30.305567 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--webserver_interface=127.25.14.62
--webserver_port=45589
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:30.462990 15147 heartbeater.cc:499] Master 127.25.14.62:41219 was elected leader, sending a full tablet report...
W20260812 19:56:30.465082 15147 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111)
W20260812 19:56:30.697971 15327 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:30.698324 15327 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:30.698431 15327 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:30.707222 15327 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:30.707345 15327 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:30.707412 15327 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:30.707484 15327 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:30.718986 15327 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=45589
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:30.720738 15327 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:30.722183 15327 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:30.731565 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:30.731534 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:30.732227 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:30.732514 15327 server_base.cc:1061] running on GCE node
I20260812 19:56:30.733492 15327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:30.735118 15327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:30.736368 15327 hybrid_clock.cc:648] HybridClock initialized: now 1786564590736321 us; error 65 us; skew 500 ppm
I20260812 19:56:30.736775 15327 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564590, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:30.740952 15327 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:30.742959 15327 webserver.cc:533] Webserver started at http://127.25.14.62:45589/ using document root <none> and password file <none>
I20260812 19:56:30.743765 15327 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:30.743916 15327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:30.751317 15327 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 19:56:30.754760 15349 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:30.756414 15327 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:56:30.756683 15327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fb726480930f458aa535cda859043cc8"
format_stamp: "Formatted at 2026-08-12 19:56:16 on dist-test-slave-p3t5"
I20260812 19:56:30.757540 15327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:30.778337 15327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:30.782887 15327 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:30.783232 15327 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260812 19:56:30.797593 14981 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41219: connect: Connection refused (error 111)
I20260812 19:56:30.808168 15327 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41219
I20260812 19:56:30.808192 15413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41219 every 8 connection(s)
I20260812 19:56:30.810276 15327 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:30.817976 15414 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:30.818161 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15327
I20260812 19:56:30.818519 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14817
I20260812 19:56:30.823096 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap starting.
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:30.840868 14972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:30.845018 15414 log.cc:826] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:30.855389 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:30.855983 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: Bootstrap complete.
I20260812 19:56:30.863718 15414 raft_consensus.cc:359] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:30.864545 15414 raft_consensus.cc:740] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb726480930f458aa535cda859043cc8, State: Initialized, Role: FOLLOWER
I20260812 19:56:30.865476 15414 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:30.866606 15421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:30.866928 15421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:30.867888 15414 sys_catalog.cc:565] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:30.886430 15435 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:30.886672 15435 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: loading cluster ID for follower catalog manager: success
I20260812 19:56:30.891244 15435 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: acquiring CA information for follower catalog manager: success
I20260812 19:56:30.894606 15435 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:56:30 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564590, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:30.941351 15327 master_runner.cc:264] No leader master found from master fb726480930f458aa535cda859043cc8
I20260812 19:56:30.941524 15327 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:30.958076 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14817
I20260812 19:56:30.987031 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40223
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43141
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:31.133361 15313 heartbeater.cc:499] Master 127.25.14.62:41219 was elected leader, sending a full tablet report...
I20260812 19:56:31.144922 15368 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8f28d67541994e998f6d8955c79f1144" instance_seqno: 1786564590066742) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:33255; Asking this server to re-register.
I20260812 19:56:31.146116 15313 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:31.148138 15368 ts_manager.cc:194] Registered new tserver with Master: 8f28d67541994e998f6d8955c79f1144 (127.25.14.3:40281)
W20260812 19:56:31.377542 15441 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:31.378106 15441 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:31.378232 15441 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:31.387360 15441 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:31.387575 15441 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:31.399277 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43141
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:31.401602 15441 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:31.403373 15441 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:31.414386 15447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:31.414515 15448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:31.415676 15450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:31.416426 15441 server_base.cc:1061] running on GCE node
I20260812 19:56:31.417078 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:31.418164 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:31.419415 15441 hybrid_clock.cc:648] HybridClock initialized: now 1786564591419361 us; error 46 us; skew 500 ppm
I20260812 19:56:31.419772 15441 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:31 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564591, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:31.423851 15441 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:31.425815 15441 webserver.cc:533] Webserver started at http://127.25.14.1:43141/ using document root <none> and password file <none>
I20260812 19:56:31.426695 15441 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:31.426854 15441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:31.429793 15457 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fb726480930f458aa535cda859043cc8)
I20260812 19:56:31.430044 15457 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:31.432075 15457 leader_election.cc:290] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb726480930f458aa535cda859043cc8 (127.25.14.62:41219), 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895)
I20260812 19:56:31.433655 15441 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 19:56:31.433751 14642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961" is_pre_election: true
I20260812 19:56:31.437022 15461 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:31.439015 15441 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:31.439520 15381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fb726480930f458aa535cda859043cc8" is_pre_election: true
I20260812 19:56:31.439234 15441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:31.440006 15381 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 2.
I20260812 19:56:31.440109 15441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:31.440711 14725 leader_election.cc:304] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7590fe7dbe474deaa8d60be3a0f85d56, fb726480930f458aa535cda859043cc8; no voters: 97dce1e214794c5cb32c247afeac6961
I20260812 19:56:31.441183 15457 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:56:31.441336 15457 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader fb726480930f458aa535cda859043cc8)
I20260812 19:56:31.441458 15457 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:31.443686 15457 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:31.444361 15457 leader_election.cc:290] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 3 election: Requested vote from peers fb726480930f458aa535cda859043cc8 (127.25.14.62:41219), 97dce1e214794c5cb32c247afeac6961 (127.25.14.61:43895)
I20260812 19:56:31.445048 14642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "97dce1e214794c5cb32c247afeac6961"
I20260812 19:56:31.445029 15381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fb726480930f458aa535cda859043cc8"
I20260812 19:56:31.445283 15381 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:31.451329 15381 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in term 3.
I20260812 19:56:31.451905 14725 leader_election.cc:304] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7590fe7dbe474deaa8d60be3a0f85d56, fb726480930f458aa535cda859043cc8; no voters: 97dce1e214794c5cb32c247afeac6961
I20260812 19:56:31.452301 15457 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:56:31.452836 15457 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [term 3 LEADER]: Becoming Leader. State: Replica: 7590fe7dbe474deaa8d60be3a0f85d56, State: Running, Role: LEADER
I20260812 19:56:31.453291 15457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:31.454968 15441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:31.455281 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.455586 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:31.456382 15469 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:31.458801 15469 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:31.459551 15441 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:31.459677 15469 catalog_manager.cc:1295] Loaded cluster ID: aa4461e3e7ad4c948e72d33974d57604
I20260812 19:56:31.459782 15469 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:31.459901 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:31.461160 15441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:31.461191 15469 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:31.462082 15469 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56: Loaded TSK: 0
I20260812 19:56:31.463161 15469 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:56:31.463240 15441 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:31.463394 15441 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:31.463536 15441 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:31.463624 15441 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:31.477133 15147 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:31.477420 15147 heartbeater.cc:499] Master 127.25.14.62:41219 was elected leader, sending a full tablet report...
I20260812 19:56:31.478643 15368 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1dab731b68f1479fa19aa02c810c1a16" instance_seqno: 1786564589390833) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:60377; Asking this server to re-register.
I20260812 19:56:31.479455 15147 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:31.480859 15368 ts_manager.cc:194] Registered new tserver with Master: 1dab731b68f1479fa19aa02c810c1a16 (127.25.14.2:41377)
I20260812 19:56:31.505208 15441 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40223
I20260812 19:56:31.505306 15597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40223 every 8 connection(s)
I20260812 19:56:31.507073 15441 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:31.511682 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15441
I20260812 19:56:31.536314 15605 raft_consensus.cc:493] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fb726480930f458aa535cda859043cc8)
I20260812 19:56:31.536827 15605 raft_consensus.cc:515] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } }
I20260812 19:56:31.539973 15605 leader_election.cc:290] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb726480930f458aa535cda859043cc8 (127.25.14.62:41219), 7590fe7dbe474deaa8d60be3a0f85d56 (127.25.14.60:42033)
Aug 12 19:56:31 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564591, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:31 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564591, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:31.561008 15599 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41219
I20260812 19:56:31.561486 15599 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:31.562866 15599 heartbeater.cc:507] Master 127.25.14.62:41219 requested a full tablet report, sending...
I20260812 19:56:31.565110 15368 ts_manager.cc:194] Registered new tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
Aug 12 19:56:31 dist-test-slave-p3t5 krb5kdc[14460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564591, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:31.566516 15598 heartbeater.cc:344] Connected to a master server at 127.25.14.60:42033
I20260812 19:56:31.566702 15598 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:31.567369 15598 heartbeater.cc:507] Master 127.25.14.60:42033 requested a full tablet report, sending...
I20260812 19:56:31.569016 14742 ts_manager.cc:194] Re-registered known tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
I20260812 19:56:31.571527 14742 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:40751
I20260812 19:56:31.574726 14754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97dce1e214794c5cb32c247afeac6961" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" is_pre_election: true
I20260812 19:56:31.577334 15600 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43895
I20260812 19:56:31.577533 15600 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:31.577975 15600 heartbeater.cc:507] Master 127.25.14.61:43895 requested a full tablet report, sending...
I20260812 19:56:31.578804 15381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97dce1e214794c5cb32c247afeac6961" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fb726480930f458aa535cda859043cc8" is_pre_election: true
I20260812 19:56:31.579182 15381 raft_consensus.cc:2393] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dce1e214794c5cb32c247afeac6961 in current term 3: Already voted for candidate 7590fe7dbe474deaa8d60be3a0f85d56 in this term.
I20260812 19:56:31.579489 14631 ts_manager.cc:194] Re-registered known tserver with Master: 8e6d88c6ee62426aa2ede0236df7196e (127.25.14.1:40223)
I20260812 19:56:31.580248 14614 leader_election.cc:304] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dce1e214794c5cb32c247afeac6961; no voters: 7590fe7dbe474deaa8d60be3a0f85d56, fb726480930f458aa535cda859043cc8
I20260812 19:56:31.580703 15605 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:31.583225 15605 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:56:31.612654 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15441
W20260812 19:56:31.614043 15618 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:31.610112 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.25.14.1:40223)
0812 19:56:31.610864 (+ 752us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:31.610873 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:31.610908 (+ 35us) server_negotiation.cc:408] Connection header received
0812 19:56:31.610976 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:31.610982 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:31.611205 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:31.611401 (+ 196us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:31.612542 (+ 1141us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34488: BlockingRecv error: recv got EOF from 127.0.0.1:34488 (error 108)
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":365,"threads_started":1}
W20260812 19:56:31.615212 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15441: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40223: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:31.615362 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15441
I20260812 19:56:31.639334 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40223
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43141
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:31.868621 15381 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [term 3 FOLLOWER]: Refusing update from remote peer 7590fe7dbe474deaa8d60be3a0f85d56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:31.869701 15466 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:31.874508 15621 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.874909 15621 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:31.876937 15621 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.877194 15621 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb726480930f458aa535cda859043cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:31.887894 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.888186 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:31.888950 14642 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [term 3 FOLLOWER]: Refusing update from remote peer 7590fe7dbe474deaa8d60be3a0f85d56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:31.889986 15457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:31.899539 15631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.899633 15605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590fe7dbe474deaa8d60be3a0f85d56. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:31.899902 15631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:31.899995 15605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97dce1e214794c5cb32c247afeac6961 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:32.063449 15620 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:32.063862 15620 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:32.064025 15620 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:32.064121 15620 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:32.073244 15620 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:32.073537 15620 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:32.085281 15620 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43141
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:32.087219 15620 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:32.088937 15620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:32.099947 15638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:32.099912 15639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:32.100104 15641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:32.101276 15620 server_base.cc:1061] running on GCE node
I20260812 19:56:32.102007 15620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:32.103009 15620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:32.104275 15620 hybrid_clock.cc:648] HybridClock initialized: now 1786564592104191 us; error 56 us; skew 500 ppm
I20260812 19:56:32.104635 15620 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:32 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:32.108671 15620 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:32.110628 15620 webserver.cc:533] Webserver started at http://127.25.14.1:43141/ using document root <none> and password file <none>
I20260812 19:56:32.111470 15620 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:32.111598 15620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:32.117781 15620 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:32.121001 15650 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:32.122712 15620 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:32.122936 15620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:32.123752 15620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:32.145790 15620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:32.150151 15620 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:32.150527 15620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:32.151664 15620 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:32.153388 15620 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:32.153463 15620 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:32.153550 15620 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:32.153605 15620 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:32.154119 15620 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:32.154186 15620 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:32.154286 15620 fault_injection.cc:59] injecting fault for kudu (pid 15620): FLAGS_fault_before_start (process will exit)
I20260812 19:56:32.186024 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40223
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--webserver_port=43141
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:32.297884 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb726480930f458aa535cda859043cc8" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41219 } } peers { permanent_uuid: "97dce1e214794c5cb32c247afeac6961" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43895 } } peers { permanent_uuid: "7590fe7dbe474deaa8d60be3a0f85d56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 42033 } } }
I20260812 19:56:32.298177 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590fe7dbe474deaa8d60be3a0f85d56 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:32.579836 15658 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:32.580235 15658 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:32.580411 15658 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:32.589331 15658 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:32.589653 15658 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:32.601135 15658 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=43141
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41219,127.25.14.61:43895,127.25.14.60:42033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:32.603392 15658 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:32.605100 15658 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:32.616398 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:32.616438 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:32.617090 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:32.618132 15658 server_base.cc:1061] running on GCE node
I20260812 19:56:32.618804 15658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:32.619915 15658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:32.621186 15658 hybrid_clock.cc:648] HybridClock initialized: now 1786564592621137 us; error 62 us; skew 500 ppm
I20260812 19:56:32.621542 15658 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:32 dist-test-slave-p3t5 krb5kdc[14460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:32.625569 15658 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:32.627614 15658 webserver.cc:533] Webserver started at http://127.25.14.1:43141/ using document root <none> and password file <none>
I20260812 19:56:32.628473 15658 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:32.628625 15658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:32.634789 15658 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:32.637969 15674 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:32.639531 15658 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:56:32.639737 15658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8e6d88c6ee62426aa2ede0236df7196e"
format_stamp: "Formatted at 2026-08-12 19:56:19 on dist-test-slave-p3t5"
I20260812 19:56:32.640559 15658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:32.668021 15658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:32.676385 15658 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:32.676716 15658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:32.677940 15658 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:32.679713 15658 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:32.679849 15658 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:32.679983 15658 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:32.680084 15658 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:32.718005 15658 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40223
I20260812 19:56:32.718097 15803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40223 every 8 connection(s)
I20260812 19:56:32.719686 15658 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:32.730079 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15658
I20260812 19:56:32.730489 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15658
W20260812 19:56:32.747205 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15658: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40223: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:32.747397 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15658
I20260812 19:56:32.774891 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14998
W20260812 19:56:32.775717 15814 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:32.738848 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:49379 (local address 127.25.14.61:43895)
0812 19:56:32.739374 (+ 526us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:32.739382 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:32.740072 (+ 690us) server_negotiation.cc:408] Connection header received
0812 19:56:32.741553 (+ 1481us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:32.741563 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:32.741696 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:32.741800 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:32.746930 (+ 5130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:32.748210 (+ 1280us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:32.774611 (+ 26401us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:49379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
W20260812 19:56:32.775717 15809 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:32.723392 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:45793 (local address 127.25.14.60:42033)
0812 19:56:32.743401 (+ 20009us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:32.743407 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:32.743430 (+ 23us) server_negotiation.cc:408] Connection header received
0812 19:56:32.743503 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:32.743508 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:32.743591 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:32.744316 (+ 725us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:32.745955 (+ 1639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:32.749795 (+ 3840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:32.774611 (+ 24816us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:45793: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19884,"thread_start_us":19784,"threads_started":1}
W20260812 19:56:32.775725 15808 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:32.723320 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:47267 (local address 127.25.14.62:41219)
0812 19:56:32.742542 (+ 19222us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:32.742548 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:32.742568 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:56:32.742634 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:32.742640 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:32.742731 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:32.742868 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:32.746550 (+ 3682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:32.752002 (+ 5452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:32.774621 (+ 22619us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:47267: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19094,"thread_start_us":18977,"threads_started":1}
W20260812 19:56:32.778576 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14998: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:41377: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:32.778744 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14998
I20260812 19:56:32.804827 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15165
W20260812 19:56:32.808037 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15165: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:40281: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:32.808225 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15165
I20260812 19:56:32.831969 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15327
W20260812 19:56:32.834371 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15327: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:41219: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:32.834517 15808 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:32.833068 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.25.14.62:41219)
0812 19:56:32.833244 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:32.833249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:32.833267 (+ 18us) server_negotiation.cc:408] Connection header received
0812 19:56:32.833331 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:32.833336 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:32.833405 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:32.833497 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:32.834245 (+ 748us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60908: BlockingRecv error: recv got EOF from 127.0.0.1:60908 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 19:56:32.834571 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15327
I20260812 19:56:32.865895 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14593
I20260812 19:56:32.867264 14671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:32.982404 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14593
I20260812 19:56:33.012365 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14703
I20260812 19:56:33.013833 14777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:33.127601 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14703
2026-08-12T19:56:33Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/7 (19671 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/8
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:56:33 dist-test-slave-p3t5 krb5kdc[15824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:33 dist-test-slave-p3t5 krb5kdc[15824](info): set up 2 sockets
Aug 12 19:56:33 dist-test-slave-p3t5 krb5kdc[15824](info): commencing operation
krb5kdc: starting...
W20260812 19:56:35.378885 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.195s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:35 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:35Z Disabled control of system clock
2026-08-12T19:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:35Z Disabled control of system clock
2026-08-12T19:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:35Z Disabled control of system clock
2026-08-12T19:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:35Z Disabled control of system clock
2026-08-12T19:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:35.957960 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:36.331351 15852 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:36.331717 15852 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:36.331810 15852 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:36.340543 15852 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:36.340669 15852 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:36.340729 15852 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:36.340778 15852 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:36.352229 15852 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:36.353993 15852 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:36.355576 15852 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:36.364496 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:36.364575 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:36.365581 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:36.366345 15852 server_base.cc:1061] running on GCE node
I20260812 19:56:36.367252 15852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:36.369004 15852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:36.370321 15852 hybrid_clock.cc:648] HybridClock initialized: now 1786564596370168 us; error 117 us; skew 500 ppm
I20260812 19:56:36.370723 15852 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:36 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:36.375319 15852 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:36.379237 15852 webserver.cc:533] Webserver started at http://127.25.14.62:43191/ using document root <none> and password file <none>
I20260812 19:56:36.380121 15852 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:36.380350 15852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:36.380878 15852 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:36.384714 15852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:36.385921 15852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:36.398866 15852 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.006s sys 0.001s
I20260812 19:56:36.404737 15868 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:36.406971 15852 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 19:56:36.407210 15852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:36.407501 15852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:36.454473 15852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:36.459007 15852 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:36.459334 15852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:36.483403 15852 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41611
I20260812 19:56:36.483412 15928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41611 every 8 connection(s)
I20260812 19:56:36.485836 15852 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:36.491366 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15852
I20260812 19:56:36.491629 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:56:36.492014 15929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:56:36.498881 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:36 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
I20260812 19:56:36.526496 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
W20260812 19:56:36.527935 15871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
W20260812 19:56:36.529697 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:36.579098 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } attempt: 1
W20260812 19:56:36.581383 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:36.633996 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35133
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:36.649691 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } attempt: 2
W20260812 19:56:36.651975 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
I20260812 19:56:36.743252 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } attempt: 3
W20260812 19:56:36.745468 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
I20260812 19:56:36.888783 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } attempt: 4
W20260812 19:56:36.891286 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111)
W20260812 19:56:37.013239 15939 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:37.013660 15939 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:37.013751 15939 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:37.022492 15939 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:37.022609 15939 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:37.022716 15939 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:37.022809 15939 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:37.034173 15939 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:37.036036 15939 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:37.037590 15939 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:37.047206 15945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:37.047253 15944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:37.048389 15947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:37.048663 15939 server_base.cc:1061] running on GCE node
I20260812 19:56:37.049603 15939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:37.051447 15939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:37.052753 15939 hybrid_clock.cc:648] HybridClock initialized: now 1786564597052573 us; error 143 us; skew 500 ppm
I20260812 19:56:37.053149 15939 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:37.056720 15939 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:56:37.058789 15939 webserver.cc:533] Webserver started at http://127.25.14.61:38301/ using document root <none> and password file <none>
I20260812 19:56:37.059705 15939 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:37.059823 15939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:37.060235 15939 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:37.063048 15939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "19784c935a35442f9d8195a4df4f38ce"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.063947 15939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "19784c935a35442f9d8195a4df4f38ce"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.070269 15939 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 19:56:37.074393 15954 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:37.076041 15939 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:56:37.076375 15939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "19784c935a35442f9d8195a4df4f38ce"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.076615 15939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:37.109385 15939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:37.113982 15939 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:37.114315 15939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:37.137075 15939 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:35133
I20260812 19:56:37.137173 16014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:35133 every 8 connection(s)
I20260812 19:56:37.138857 15939 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:56:37.143350 16015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:56:37.147620 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15939
I20260812 19:56:37.147840 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:56:37.150830 16015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:37.176152 16015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:37.190644 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } attempt: 5
I20260812 19:56:37.197075 16015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:56:37.198432 15958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
W20260812 19:56:37.200390 16015 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:37.209755 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:56:37.211957 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:37.249758 16015 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 1
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
W20260812 19:56:37.252082 16015 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 19:56:37.263218 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 1
W20260812 19:56:37.265555 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:37.290180 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44601
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:37.320333 16015 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 2
W20260812 19:56:37.322540 16015 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
I20260812 19:56:37.333851 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 2
W20260812 19:56:37.336113 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
I20260812 19:56:37.413853 16015 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 3
W20260812 19:56:37.416464 16015 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
I20260812 19:56:37.442431 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 3
W20260812 19:56:37.445060 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
I20260812 19:56:37.559758 16015 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 4
W20260812 19:56:37.562338 16015 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
I20260812 19:56:37.622366 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 4
W20260812 19:56:37.624874 15929 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111)
W20260812 19:56:37.673379 16026 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:37.673764 16026 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:37.673859 16026 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:37.682499 16026 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:37.682636 16026 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:37.682696 16026 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:37.682746 16026 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:37.694128 16026 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:37.695907 16026 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:37.697392 16026 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:37.710103 16035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:37.709913 16032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:37.709914 16031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:37.711912 16026 server_base.cc:1061] running on GCE node
I20260812 19:56:37.713001 16026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:37.714994 16026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:37.716297 16026 hybrid_clock.cc:648] HybridClock initialized: now 1786564597716146 us; error 99 us; skew 500 ppm
I20260812 19:56:37.716719 16026 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:37.720312 16026 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:56:37.722227 16026 webserver.cc:533] Webserver started at http://127.25.14.60:41017/ using document root <none> and password file <none>
I20260812 19:56:37.723058 16026 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:37.723198 16026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:37.723619 16026 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:37.726483 16026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "b9c4be8171d04121a8c2aa7f27c50087"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.727351 16026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "b9c4be8171d04121a8c2aa7f27c50087"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.733335 16026 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:56:37.737077 16047 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:37.738768 16026 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:37.738996 16026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "b9c4be8171d04121a8c2aa7f27c50087"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:37.739230 16026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:37.758853 16026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:37.769711 16026 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:37.770061 16026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:37.795895 16026 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:44601
I20260812 19:56:37.795928 16106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:44601 every 8 connection(s)
I20260812 19:56:37.797778 16026 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:56:37.802659 16107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:56:37.804387 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16026
I20260812 19:56:37.804661 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:37.812951 16107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:37.848733 16107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:37.861668 16015 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 5
I20260812 19:56:37.864130 16107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564597, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:37.884059 16015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Bootstrap starting.
I20260812 19:56:37.885075 16107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Bootstrap starting.
I20260812 19:56:37.887598 16015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:37.888335 16107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:37.888909 16015 log.cc:826] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:37.889762 16107 log.cc:826] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:37.892572 16015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: No bootstrap required, opened a new log
I20260812 19:56:37.892731 16107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: No bootstrap required, opened a new log
I20260812 19:56:37.898074 16015 raft_consensus.cc:359] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.898125 16107 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.898414 16015 raft_consensus.cc:385] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:37.898504 16107 raft_consensus.cc:385] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:37.898535 16015 raft_consensus.cc:740] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19784c935a35442f9d8195a4df4f38ce, State: Initialized, Role: FOLLOWER
I20260812 19:56:37.898644 16107 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4be8171d04121a8c2aa7f27c50087, State: Initialized, Role: FOLLOWER
I20260812 19:56:37.899351 16015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.899506 16107 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.900974 16118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:37.901311 16118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:37.901156 16119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:37.901678 16119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:37.902143 15929 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } attempt: 5
I20260812 19:56:37.902452 16107 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:37.902468 16015 sys_catalog.cc:565] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:37.903524 16119 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:56:37.903802 16119 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.909137 15988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19784c935a35442f9d8195a4df4f38ce" is_pre_election: true
I20260812 19:56:37.909971 15988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c4be8171d04121a8c2aa7f27c50087 in term 0.
Aug 12 19:56:37 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:37.914078 16052 leader_election.cc:304] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, b9c4be8171d04121a8c2aa7f27c50087; no voters:
I20260812 19:56:37.922844 16119 leader_election.cc:290] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611), 19784c935a35442f9d8195a4df4f38ce (127.25.14.61:35133)
I20260812 19:56:37.924470 16119 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:56:37.925451 16119 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:56:37.930608 16119 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:37.934382 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap starting.
I20260812 19:56:37.939931 15929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Neither blocks nor log segments found. Creating new log.
I20260812 19:56:37.942224 15929 log.cc:826] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:37.947167 16119 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.949694 15988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19784c935a35442f9d8195a4df4f38ce"
I20260812 19:56:37.950052 15988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:37.956213 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: No bootstrap required, opened a new log
I20260812 19:56:37.958844 15988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4be8171d04121a8c2aa7f27c50087 in term 1.
I20260812 19:56:37.959503 16052 leader_election.cc:304] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, b9c4be8171d04121a8c2aa7f27c50087; no voters:
I20260812 19:56:37.960021 16123 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:56:37.963434 16119 leader_election.cc:290] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 1 election: Requested vote from peers de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611), 19784c935a35442f9d8195a4df4f38ce (127.25.14.61:35133)
I20260812 19:56:37.969702 16123 raft_consensus.cc:697] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 LEADER]: Becoming Leader. State: Replica: b9c4be8171d04121a8c2aa7f27c50087, State: Running, Role: LEADER
I20260812 19:56:37.971068 16123 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.974056 15929 raft_consensus.cc:359] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.974705 15929 raft_consensus.cc:385] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:56:37.974928 15929 raft_consensus.cc:740] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de82d3b77b4449468e9cbe58c1180bbb, State: Initialized, Role: FOLLOWER
I20260812 19:56:37.976089 15929 consensus_queue.cc:260] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:37.978734 16119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c4be8171d04121a8c2aa7f27c50087. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:37.979076 16119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:37.992345 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:37.992715 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:37.994268 15929 sys_catalog.cc:565] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:37.994529 15902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de82d3b77b4449468e9cbe58c1180bbb"
I20260812 19:56:37.983372 16026 master_runner.cc:187] Error getting master registration for 127.25.14.62:41611: OK, instance_id { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" instance_seqno: 1786564596475505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:56:37.995064 16026 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:37.995110 15902 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:56:37.997460 15903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de82d3b77b4449468e9cbe58c1180bbb" is_pre_election: true
I20260812 19:56:37.999207 15902 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4be8171d04121a8c2aa7f27c50087 in term 1.
I20260812 19:56:37.998564 15939 master_runner.cc:187] Error getting master registration for 127.25.14.62:41611: OK, instance_id { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" instance_seqno: 1786564596475505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:56:38.005863 15939 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:56:38.007930 16147 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:38.008113 16147 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:56:38.008260 16148 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:38.010666 16148 catalog_manager.cc:1520] Initializing Kudu cluster ID...
W20260812 19:56:38.011008 16149 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:38.011152 16149 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:56:38.022401 16055 tablet.cc:2406] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:56:38.032723 15988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Refusing update from remote peer b9c4be8171d04121a8c2aa7f27c50087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:56:38.033779 15902 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Refusing update from remote peer b9c4be8171d04121a8c2aa7f27c50087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:56:38.033926 16119 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:56:38.039414 16119 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:56:38.049948 16118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c4be8171d04121a8c2aa7f27c50087. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.050258 16118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:38.061324 16167 mvcc.cc:204] Tried to move back new op lower bound from 7317768593524486144 to 7317768593296433152. Current Snapshot: MvccSnapshot[applied={T|T < 7317768593524486144}]
I20260812 19:56:38.063382 16157 mvcc.cc:204] Tried to move back new op lower bound from 7317768593524486144 to 7317768593296433152. Current Snapshot: MvccSnapshot[applied={T|T < 7317768593524486144}]
I20260812 19:56:38.066952 16118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.067219 16118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:38.070459 16136 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.070853 16136 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:38.072000 16166 catalog_manager.cc:1594] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:56:38.072136 16166 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:56:38.072655 16148 catalog_manager.cc:1383] Generated new cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:38.072764 16148 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:38.073470 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c4be8171d04121a8c2aa7f27c50087. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.073747 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:38.082125 16136 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.082408 16136 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:38.094339 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:38.094614 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:38.103523 16148 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:56:38.109802 16148 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:38.147132 16148 catalog_manager.cc:6092] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Generated new TSK 0
I20260812 19:56:38.148478 16148 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:38.343636 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:38.875444 16179 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:38.875797 16179 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:38.875985 16179 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:38.884852 16179 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:38.885121 16179 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:38.896287 16179 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:38.898161 16179 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:38.899682 16179 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:38.910532 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:38.910774 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:38.910748 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:38.911912 16179 server_base.cc:1061] running on GCE node
I20260812 19:56:38.912559 16179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:38.913794 16179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:38.915235 16179 hybrid_clock.cc:648] HybridClock initialized: now 1786564598915026 us; error 156 us; skew 500 ppm
I20260812 19:56:38.915628 16179 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:38 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:38.919332 16179 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:38.921118 16179 webserver.cc:533] Webserver started at http://127.25.14.1:35385/ using document root <none> and password file <none>
I20260812 19:56:38.922037 16179 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:38.922147 16179 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:38.922536 16179 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:38.925137 16179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:38.926074 16179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:38.932296 16179 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:56:38.936008 16197 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:38.937676 16179 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:56:38.937914 16179 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:38.938115 16179 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:38.976280 16179 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:38.980721 16179 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:38.981045 16179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:38.982277 16179 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:38.984064 16179 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:38.984143 16179 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:38.984265 16179 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:38.984326 16179 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:39.014045 16149 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:39.014179 16149 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: loading cluster ID for follower catalog manager: success
I20260812 19:56:39.018697 16149 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: acquiring CA information for follower catalog manager: success
I20260812 19:56:39.021703 16149 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:39.022408 16179 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38719
I20260812 19:56:39.022480 16330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38719 every 8 connection(s)
I20260812 19:56:39.023938 16179 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:39.032128 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16179
I20260812 19:56:39.032373 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:39.054638 16333 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:39.054733 16332 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:39.055120 16332 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.055119 16333 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.055439 16331 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:39.055581 16331 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.056493 16333 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:39.056515 16332 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:39.058522 16331 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:39.059567 15890 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
I20260812 19:56:39.059569 15974 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
I20260812 19:56:39.060204 16068 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
I20260812 19:56:39.062114 16068 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:55427
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
I20260812 19:56:39.074352 16166 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:39.074481 16166 catalog_manager.cc:1588] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: loading cluster ID for follower catalog manager: success
I20260812 19:56:39.078636 16166 catalog_manager.cc:1610] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: acquiring CA information for follower catalog manager: success
I20260812 19:56:39.081398 16166 catalog_manager.cc:1638] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:39.176971 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:39.553480 16348 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:39.553956 16348 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:39.554064 16348 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:39.563021 16348 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:39.563234 16348 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:39.574746 16348 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:39.576531 16348 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:39.578052 16348 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:39.588682 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:39.588716 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:39.589393 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:39.590131 16348 server_base.cc:1061] running on GCE node
I20260812 19:56:39.590826 16348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:39.591984 16348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:39.593281 16348 hybrid_clock.cc:648] HybridClock initialized: now 1786564599593113 us; error 131 us; skew 500 ppm
I20260812 19:56:39.593676 16348 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:39.597266 16348 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:39.599051 16348 webserver.cc:533] Webserver started at http://127.25.14.2:46875/ using document root <none> and password file <none>
I20260812 19:56:39.599835 16348 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:39.599973 16348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:39.600358 16348 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:39.603101 16348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "49396012b74d4831b8a388f967c9d707"
format_stamp: "Formatted at 2026-08-12 19:56:39 on dist-test-slave-p3t5"
I20260812 19:56:39.604125 16348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "49396012b74d4831b8a388f967c9d707"
format_stamp: "Formatted at 2026-08-12 19:56:39 on dist-test-slave-p3t5"
I20260812 19:56:39.610159 16348 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 19:56:39.614073 16365 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:39.615757 16348 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:56:39.615972 16348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "49396012b74d4831b8a388f967c9d707"
format_stamp: "Formatted at 2026-08-12 19:56:39 on dist-test-slave-p3t5"
I20260812 19:56:39.616178 16348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:39.640344 16348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:39.644394 16348 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:39.644778 16348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:39.645980 16348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:39.647677 16348 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:39.647751 16348 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:39.647837 16348 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:39.647883 16348 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:39.684484 16348 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:43771
I20260812 19:56:39.684578 16498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:43771 every 8 connection(s)
I20260812 19:56:39.686036 16348 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:39.690121 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16348
I20260812 19:56:39.690399 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:39.717434 16501 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:39.717940 16501 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.719177 16501 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:39.721468 15974 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:43771)
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:39 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:39.732216 16499 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:39.732431 16499 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.733048 16499 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:39.733150 16500 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:39.733330 16500 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:39.733985 16500 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:39.734238 16068 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:43771)
I20260812 19:56:39.735110 16068 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:42531
I20260812 19:56:39.735230 15890 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:43771)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:39.850134 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:40.064670 16331 heartbeater.cc:499] Master 127.25.14.60:44601 was elected leader, sending a full tablet report...
W20260812 19:56:40.234375 16513 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:40.234743 16513 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:40.234912 16513 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:40.243624 16513 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:40.243880 16513 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:40.255429 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:40.257273 16513 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:40.258816 16513 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:40.270313 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:40.270246 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:40.270272 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:40.271770 16513 server_base.cc:1061] running on GCE node
I20260812 19:56:40.272413 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:40.273699 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:40.275099 16513 hybrid_clock.cc:648] HybridClock initialized: now 1786564600274941 us; error 119 us; skew 500 ppm
I20260812 19:56:40.275473 16513 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:40 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:40.279313 16513 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:40.281391 16513 webserver.cc:533] Webserver started at http://127.25.14.3:36959/ using document root <none> and password file <none>
I20260812 19:56:40.282539 16513 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:40.282696 16513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:40.283125 16513 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:56:40.286067 16513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "b66bd38d1f31472bb23a900f36240af1"
format_stamp: "Formatted at 2026-08-12 19:56:40 on dist-test-slave-p3t5"
I20260812 19:56:40.286980 16513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "b66bd38d1f31472bb23a900f36240af1"
format_stamp: "Formatted at 2026-08-12 19:56:40 on dist-test-slave-p3t5"
I20260812 19:56:40.293082 16513 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 19:56:40.297029 16532 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:40.298548 16513 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:56:40.298763 16513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "b66bd38d1f31472bb23a900f36240af1"
format_stamp: "Formatted at 2026-08-12 19:56:40 on dist-test-slave-p3t5"
I20260812 19:56:40.298995 16513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:40.316977 16513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:40.321072 16513 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:40.321453 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:40.322621 16513 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:40.324555 16513 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:40.324635 16513 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:40.324800 16513 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:40.324883 16513 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:40.363060 16513 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:34963
I20260812 19:56:40.363103 16656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:34963 every 8 connection(s)
I20260812 19:56:40.364616 16513 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:40.374091 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16513
I20260812 19:56:40.374328 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:40 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:40 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:56:40 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:40.394421 16659 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:40.394719 16657 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:40.394630 16658 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:40.394970 16658 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:40.394972 16657 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:40.394968 16659 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:40.396504 16659 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:40.396544 16658 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:40.396793 16657 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:40.398989 16068 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:34963)
I20260812 19:56:40.399070 15890 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:34963)
I20260812 19:56:40.399220 15974 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:34963)
I20260812 19:56:40.400189 16068 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:40777
I20260812 19:56:40.403899 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:40.412155 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:40.412377 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15852
I20260812 19:56:40.413341 15924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:40.525907 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15852
I20260812 19:56:40.555696 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--webserver_interface=127.25.14.62
--webserver_port=43191
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:40.737982 16500 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111)
I20260812 19:56:40.738248 16499 heartbeater.cc:499] Master 127.25.14.60:44601 was elected leader, sending a full tablet report...
W20260812 19:56:40.754971 16052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111)
W20260812 19:56:40.756691 16052 consensus_peers.cc:597] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 -> Peer de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611): Couldn't send request to peer de82d3b77b4449468e9cbe58c1180bbb. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:56:40.952111 16668 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:40.952540 16668 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:40.952701 16668 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:40.961462 16668 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:40.961655 16668 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:40.961786 16668 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:40.961871 16668 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:40.973742 16668 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=43191
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:40.975638 16668 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:40.977186 16668 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:40.989681 16680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:40.989681 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:40.991350 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:40.991575 16668 server_base.cc:1061] running on GCE node
I20260812 19:56:40.992529 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:40.994421 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:40.995810 16668 hybrid_clock.cc:648] HybridClock initialized: now 1786564600995642 us; error 128 us; skew 500 ppm
I20260812 19:56:40.996229 16668 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:40 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:41.000007 16668 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:41.001968 16668 webserver.cc:533] Webserver started at http://127.25.14.62:43191/ using document root <none> and password file <none>
I20260812 19:56:41.002823 16668 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:41.002979 16668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:41.009507 16668 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:41.012951 16692 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:41.014744 16668 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:56:41.014964 16668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:41.015806 16668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:41.048082 16668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:41.052637 16668 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:41.052959 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260812 19:56:41.065820 16332 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111)
I20260812 19:56:41.076810 16668 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41611
I20260812 19:56:41.076833 16754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41611 every 8 connection(s)
I20260812 19:56:41.078715 16668 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:41.080974 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16668
I20260812 19:56:41.081390 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16179
I20260812 19:56:41.087852 16755 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:41.093210 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap starting.
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:41.102921 16326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:41.115742 16755 log.cc:826] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:41.126170 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:41.126866 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap complete.
I20260812 19:56:41.134495 16755 raft_consensus.cc:359] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:41.135288 16755 raft_consensus.cc:740] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de82d3b77b4449468e9cbe58c1180bbb, State: Initialized, Role: FOLLOWER
I20260812 19:56:41.136204 16755 consensus_queue.cc:260] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:41.137252 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:41.137718 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:41.139369 16755 sys_catalog.cc:565] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:41.159951 16773 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:41.160208 16773 catalog_manager.cc:1588] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: loading cluster ID for follower catalog manager: success
I20260812 19:56:41.164614 16773 catalog_manager.cc:1610] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: acquiring CA information for follower catalog manager: success
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564600, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:41.167920 16773 catalog_manager.cc:1638] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:41.191969 16119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:41.192420 16119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:41.200675 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c4be8171d04121a8c2aa7f27c50087. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:41.200955 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:41.224236 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16179
I20260812 19:56:41.250962 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38719
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=35385
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:41.402925 16657 heartbeater.cc:499] Master 127.25.14.60:44601 was elected leader, sending a full tablet report...
I20260812 19:56:41.412884 16713 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b66bd38d1f31472bb23a900f36240af1" instance_seqno: 1786564600354233) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:40687; Asking this server to re-register.
I20260812 19:56:41.413936 16658 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:41.415855 16713 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:34963)
W20260812 19:56:41.643807 16783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:41.644157 16783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:41.644289 16783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:41.653352 16783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:41.653659 16783 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:41.664899 16783 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35385
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:41.666795 16783 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:41.668293 16783 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:41.678862 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:41.678910 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:41.678825 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:41.679898 16783 server_base.cc:1061] running on GCE node
I20260812 19:56:41.680576 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:41.681944 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:41.683411 16783 hybrid_clock.cc:648] HybridClock initialized: now 1786564601683246 us; error 118 us; skew 500 ppm
I20260812 19:56:41.683763 16783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564601, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:41.692011 16783 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:41.694106 16783 webserver.cc:533] Webserver started at http://127.25.14.1:35385/ using document root <none> and password file <none>
I20260812 19:56:41.694957 16783 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:41.695135 16783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:41.701529 16783 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:56:41.704782 16800 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:41.706578 16783 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:41.706811 16783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:41.707592 16783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:41.723136 16783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:41.727751 16783 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:41.728013 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:41.728998 16783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:41.730759 16783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:41.730858 16783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:41.730945 16783 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:41.731002 16783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:41.748004 16500 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:41.749173 16713 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "49396012b74d4831b8a388f967c9d707" instance_seqno: 1786564599675389) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:60011; Asking this server to re-register.
I20260812 19:56:41.749768 16500 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:41.751289 16713 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:43771)
I20260812 19:56:41.769804 16783 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38719
I20260812 19:56:41.769861 16932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38719 every 8 connection(s)
I20260812 19:56:41.771562 16783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:41.772536 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9c4be8171d04121a8c2aa7f27c50087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:41.772874 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:41.777551 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16783
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564601, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564601, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:41.818514 16934 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:41.818941 16934 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:41.820050 16934 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:41.821748 16936 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:41.821915 16936 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:41.822602 16068 ts_manager.cc:194] Re-registered known tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
I20260812 19:56:41.823786 16068 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:60549
I20260812 19:56:41.824052 16936 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:41.825608 16713 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
Aug 12 19:56:41 dist-test-slave-p3t5 krb5kdc[15824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564601, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:41.835340 16941 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:41.835539 16941 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:41.836004 16941 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:41.837426 15974 ts_manager.cc:194] Re-registered known tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:38719)
I20260812 19:56:41.852378 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16783
W20260812 19:56:41.853696 16949 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:41.849733 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.25.14.1:38719)
0812 19:56:41.850712 (+ 979us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:41.850719 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:41.850750 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:56:41.850816 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:41.850823 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:41.851051 (+ 228us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:41.851240 (+ 189us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:41.852272 (+ 1032us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60112: BlockingRecv error: recv got EOF from 127.0.0.1:60112 (error 108)
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":589,"threads_started":1}
W20260812 19:56:41.854903 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16783: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:38719: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:41.855054 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16783
I20260812 19:56:41.879144 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38719
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=35385
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:42.291497 16951 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:42.291913 16951 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:42.292078 16951 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:42.292171 16951 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:42.300925 16951 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:42.301221 16951 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:42.313136 16951 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35385
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:42.314975 16951 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:42.316663 16951 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:42.327687 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:42.327661 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:42.327718 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:42.328713 16951 server_base.cc:1061] running on GCE node
I20260812 19:56:42.329380 16951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:42.330686 16951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:42.332070 16951 hybrid_clock.cc:648] HybridClock initialized: now 1786564602331870 us; error 144 us; skew 500 ppm
I20260812 19:56:42.332427 16951 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:42 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564602, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:42.336072 16951 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:42.338027 16951 webserver.cc:533] Webserver started at http://127.25.14.1:35385/ using document root <none> and password file <none>
I20260812 19:56:42.338809 16951 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:42.338903 16951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:42.344771 16951 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:42.347720 16968 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:42.349418 16951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:56:42.349606 16951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:42.350510 16951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:42.380285 16951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:42.384928 16951 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:42.385210 16951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:42.386255 16951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:42.388092 16951 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:42.388170 16951 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:42.388259 16951 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:42.388303 16951 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:42.388630 16951 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:42.388695 16951 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:42.388804 16951 fault_injection.cc:59] injecting fault for kudu (pid 16951): FLAGS_fault_before_start (process will exit)
I20260812 19:56:42.413552 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38719
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=35385
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:42.799885 16978 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:42.800341 16978 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:42.800496 16978 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:42.812032 16978 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:42.812251 16978 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:42.823546 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36983,127.25.14.21:39785,127.25.14.22:33975,127.25.14.23:42891,127.25.14.24:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35385
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:42.825367 16978 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:42.826876 16978 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:42.838029 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:42.838049 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:42.838029 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:42.839360 16978 server_base.cc:1061] running on GCE node
I20260812 19:56:42.840023 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:42.841327 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:42.842597 16978 hybrid_clock.cc:648] HybridClock initialized: now 1786564602842409 us; error 148 us; skew 500 ppm
I20260812 19:56:42.843043 16978 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:42 dist-test-slave-p3t5 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564602, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:42.847357 16978 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:42.849426 16978 webserver.cc:533] Webserver started at http://127.25.14.1:35385/ using document root <none> and password file <none>
I20260812 19:56:42.850382 16978 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:42.850545 16978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:42.857080 16978 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:56:42.860235 16993 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:42.861886 16978 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:56:42.862085 16978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:42.862882 16978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:42.886910 16978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:42.894613 16978 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:42.894940 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:42.896021 16978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:42.897895 16978 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:42.898018 16978 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:42.898152 16978 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:42.898247 16978 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:42.935670 16978 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38719
I20260812 19:56:42.935734 17127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38719 every 8 connection(s)
I20260812 19:56:42.937495 16978 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:42.938720 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16978
I20260812 19:56:42.939143 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16978
W20260812 19:56:42.954857 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16978: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:38719: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:42.955066 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16978
I20260812 19:56:42.978971 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16348
W20260812 19:56:42.979795 17142 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:42.951500 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48695 (local address 127.25.14.62:41611)
0812 19:56:42.955999 (+ 4499us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:42.957018 (+ 1019us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:42.957045 (+ 27us) server_negotiation.cc:408] Connection header received
0812 19:56:42.978773 (+ 21728us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48695: BlockingRecv error: recv got EOF from 127.25.14.1:48695 (error 108)
Metrics: {"server-negotiator.queue_time_us":4344,"thread_start_us":2917,"threads_started":1}
W20260812 19:56:42.979789 17140 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:42.950620 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:44659 (local address 127.25.14.61:35133)
0812 19:56:42.951961 (+ 1341us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:42.951971 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:42.951998 (+ 27us) server_negotiation.cc:408] Connection header received
0812 19:56:42.953495 (+ 1497us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:42.953505 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:42.953645 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:42.957582 (+ 3937us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:42.978808 (+ 21226us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:44659: BlockingRecv error: recv got EOF from 127.25.14.1:44659 (error 108)
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":1017,"threads_started":1}
W20260812 19:56:42.979808 17134 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:42.942223 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:46935 (local address 127.25.14.60:44601)
0812 19:56:42.948612 (+ 6389us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:42.948620 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:42.951571 (+ 2951us) server_negotiation.cc:408] Connection header received
0812 19:56:42.953506 (+ 1935us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:42.953514 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:42.957214 (+ 3700us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:42.957344 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:42.978720 (+ 21376us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:46935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":6145,"thread_start_us":5980,"threads_started":1}
W20260812 19:56:42.982925 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16348: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:43771: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:42.983100 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16348
I20260812 19:56:43.007237 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16513
W20260812 19:56:43.010563 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16513: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:34963: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:43.010758 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16513
I20260812 19:56:43.036226 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16668
W20260812 19:56:43.038683 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16668: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:41611: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:43.038820 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16668
W20260812 19:56:43.038848 17142 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:43.037357 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.25.14.62:41611)
0812 19:56:43.037554 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:43.037559 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:43.037579 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:56:43.037689 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:43.037694 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:43.037768 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:43.037854 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:43.038591 (+ 737us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40906: BlockingRecv error: recv got EOF from 127.0.0.1:40906 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 19:56:43.068327 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15939
I20260812 19:56:43.069723 16010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:43.183540 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15939
I20260812 19:56:43.210716 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16026
I20260812 19:56:43.211968 16102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:43.326622 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16026
2026-08-12T19:56:43Z chronyd exiting
2026-08-12T19:56:43Z chronyd exiting
2026-08-12T19:56:43Z chronyd exiting
2026-08-12T19:56:43Z chronyd exiting
2026-08-12T19:56:43Z chronyd exiting
Aug 12 19:56:43 dist-test-slave-p3t5 krb5kdc[17150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:43 dist-test-slave-p3t5 krb5kdc[17150](info): set up 2 sockets
Aug 12 19:56:43 dist-test-slave-p3t5 krb5kdc[17150](info): commencing operation
krb5kdc: starting...
W20260812 19:56:45.508486 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.095s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:45 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:56:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:45Z Disabled control of system clock
2026-08-12T19:56:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:45Z Disabled control of system clock
2026-08-12T19:56:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:45Z Disabled control of system clock
2026-08-12T19:56:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:45Z Disabled control of system clock
2026-08-12T19:56:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:56:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:46.052968 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:46.431092 17175 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:46.431511 17175 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:46.431671 17175 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:46.440327 17175 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:46.440485 17175 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:46.440610 17175 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:46.440693 17175 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:46.452473 17175 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:46.454445 17175 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:46.456038 17175 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:46.465219 17182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:46.465230 17181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:46.466101 17184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:46.466884 17175 server_base.cc:1061] running on GCE node
I20260812 19:56:46.467813 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:46.469663 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:46.471079 17175 hybrid_clock.cc:648] HybridClock initialized: now 1786564606470907 us; error 130 us; skew 500 ppm
I20260812 19:56:46.471515 17175 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:46 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564606, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:46.475322 17175 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:46.477181 17175 webserver.cc:533] Webserver started at http://127.25.14.62:45195/ using document root <none> and password file <none>
I20260812 19:56:46.478103 17175 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:46.478256 17175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:46.484221 17175 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 19:56:46.487643 17191 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:46.489142 17175 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 19:56:46.489352 17175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:46.490182 17175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:46.511799 17175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:46.515829 17175 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:46.516129 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:46.538298 17175 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41611
I20260812 19:56:46.538338 17251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41611 every 8 connection(s)
I20260812 19:56:46.540215 17175 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:46.546766 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17175
I20260812 19:56:46.547008 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:56:46.548038 17253 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:46.551702 17253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap starting.
I20260812 19:56:46.572306 17253 log.cc:826] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:46.583529 17253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:46.584233 17253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap complete.
I20260812 19:56:46.593926 17253 raft_consensus.cc:359] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:46.594695 17253 raft_consensus.cc:740] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de82d3b77b4449468e9cbe58c1180bbb, State: Initialized, Role: FOLLOWER
I20260812 19:56:46.595641 17253 consensus_queue.cc:260] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:46.596856 17258 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:46.597182 17258 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:46.598198 17253 sys_catalog.cc:565] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:46.618255 17269 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:46.618432 17269 catalog_manager.cc:1588] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: loading cluster ID for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:46.622937 17269 catalog_manager.cc:1610] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: acquiring CA information for follower catalog manager: success
I20260812 19:56:46.625857 17269 catalog_manager.cc:1638] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:56:46 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564606, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:46.643488 17175 master_runner.cc:187] Error getting master registration for 127.25.14.61:35133: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35133: connect: Connection refused (error 111),
I20260812 19:56:46.647321 17175 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:46.701776 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35133
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:47.114590 17275 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:47.114984 17275 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:47.115141 17275 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:47.123936 17275 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:47.124061 17275 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:47.124166 17275 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:47.124238 17275 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:47.135682 17275 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:47.137482 17275 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:47.139092 17275 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:47.147861 17282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:47.148141 17281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:47.148300 17285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:47.149255 17275 server_base.cc:1061] running on GCE node
I20260812 19:56:47.150188 17275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:47.151902 17275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:47.153153 17275 hybrid_clock.cc:648] HybridClock initialized: now 1786564607153001 us; error 115 us; skew 500 ppm
I20260812 19:56:47.153535 17275 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564607, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:47.157235 17275 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:56:47.159121 17275 webserver.cc:533] Webserver started at http://127.25.14.61:33333/ using document root <none> and password file <none>
I20260812 19:56:47.159941 17275 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:47.160110 17275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:47.166332 17275 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:47.169945 17292 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:47.171540 17275 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:56:47.171765 17275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "19784c935a35442f9d8195a4df4f38ce"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:47.172602 17275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:47.199285 17275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:47.203823 17275 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:47.204135 17275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:47.227028 17275 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:35133
I20260812 19:56:47.227041 17344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:35133 every 8 connection(s)
I20260812 19:56:47.228873 17275 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:56:47.231014 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17275
I20260812 19:56:47.231274 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:56:47.236799 17345 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:56:47.240540 17345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Bootstrap starting.
I20260812 19:56:47.259130 17345 log.cc:826] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:47.268818 17345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:47.269452 17345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Bootstrap complete.
I20260812 19:56:47.278246 17345 raft_consensus.cc:359] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:47.279410 17345 raft_consensus.cc:740] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19784c935a35442f9d8195a4df4f38ce, State: Initialized, Role: FOLLOWER
I20260812 19:56:47.280426 17345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:47.281436 17352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:47.281812 17352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:47.283123 17345 sys_catalog.cc:565] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:47.297186 17366 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:47.297374 17366 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: loading cluster ID for follower catalog manager: success
I20260812 19:56:47.302304 17366 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: acquiring CA information for follower catalog manager: success
I20260812 19:56:47.304704 17366 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564607, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:47.336409 17275 master_runner.cc:187] Error getting master registration for 127.25.14.60:44601: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44601: connect: Connection refused (error 111),
I20260812 19:56:47.340305 17275 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:47.388813 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44601
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:56:47.797555 17375 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:47.798022 17375 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:47.798111 17375 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:47.807813 17375 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:47.807942 17375 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:47.807999 17375 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:47.808041 17375 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:47.820176 17375 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:47.821940 17375 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:47.823393 17375 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:47.838690 17381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:47.839359 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:47.839522 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:47.840370 17375 server_base.cc:1061] running on GCE node
I20260812 19:56:47.841248 17375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:47.843138 17375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:47.844409 17375 hybrid_clock.cc:648] HybridClock initialized: now 1786564607844264 us; error 108 us; skew 500 ppm
I20260812 19:56:47.844831 17375 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564607, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:47.848984 17375 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:56:47.851081 17375 webserver.cc:533] Webserver started at http://127.25.14.60:39111/ using document root <none> and password file <none>
I20260812 19:56:47.851970 17375 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:47.852145 17375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:47.858568 17375 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:47.862090 17393 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:47.863718 17375 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:56:47.863978 17375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "b9c4be8171d04121a8c2aa7f27c50087"
format_stamp: "Formatted at 2026-08-12 19:56:37 on dist-test-slave-p3t5"
I20260812 19:56:47.864816 17375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:47.903939 17375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:47.912761 17375 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:47.913117 17375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:47.936632 17375 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:44601
I20260812 19:56:47.936652 17449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:44601 every 8 connection(s)
I20260812 19:56:47.938557 17375 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:56:47.939579 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17375
I20260812 19:56:47.939832 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:56:47.952042 17450 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:47.960497 17450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Bootstrap starting.
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:47.983595 17450 log.cc:826] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Log is configured to *not* fsync() on all Append() calls
Aug 12 19:56:47 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:47.997215 17450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:47.998031 17450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: Bootstrap complete.
I20260812 19:56:48.006582 17450 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:48.007539 17450 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4be8171d04121a8c2aa7f27c50087, State: Initialized, Role: FOLLOWER
I20260812 19:56:48.008389 17450 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:48.009395 17457 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.009714 17457 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:48.010674 17450 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:48.025532 17469 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:48.025683 17469 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: loading cluster ID for follower catalog manager: success
I20260812 19:56:48.030067 17469 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: acquiring CA information for follower catalog manager: success
Aug 12 19:56:48 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564607, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:48.033052 17469 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:48.064204 17375 master_runner.cc:264] No leader master found from master b9c4be8171d04121a8c2aa7f27c50087
I20260812 19:56:48.064405 17375 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:48.176858 17472 raft_consensus.cc:493] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:56:48.177222 17472 raft_consensus.cc:515] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:48.180064 17472 leader_election.cc:290] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19784c935a35442f9d8195a4df4f38ce (127.25.14.61:35133), b9c4be8171d04121a8c2aa7f27c50087 (127.25.14.60:44601)
I20260812 19:56:48.188606 17320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de82d3b77b4449468e9cbe58c1180bbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "19784c935a35442f9d8195a4df4f38ce" is_pre_election: true
I20260812 19:56:48.189282 17320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de82d3b77b4449468e9cbe58c1180bbb in term 1.
I20260812 19:56:48.190056 17194 leader_election.cc:304] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, de82d3b77b4449468e9cbe58c1180bbb; no voters:
I20260812 19:56:48.190564 17472 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:56:48.190738 17472 raft_consensus.cc:493] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:56:48.190847 17472 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:48.190634 17424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de82d3b77b4449468e9cbe58c1180bbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9c4be8171d04121a8c2aa7f27c50087" is_pre_election: true
I20260812 19:56:48.191299 17424 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de82d3b77b4449468e9cbe58c1180bbb in term 1.
I20260812 19:56:48.195305 17472 raft_consensus.cc:515] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:48.196120 17472 leader_election.cc:290] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [CANDIDATE]: Term 2 election: Requested vote from peers 19784c935a35442f9d8195a4df4f38ce (127.25.14.61:35133), b9c4be8171d04121a8c2aa7f27c50087 (127.25.14.60:44601)
I20260812 19:56:48.196736 17320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de82d3b77b4449468e9cbe58c1180bbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "19784c935a35442f9d8195a4df4f38ce"
I20260812 19:56:48.196810 17424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de82d3b77b4449468e9cbe58c1180bbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9c4be8171d04121a8c2aa7f27c50087"
I20260812 19:56:48.196969 17320 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:48.197042 17424 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:56:48.201851 17320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de82d3b77b4449468e9cbe58c1180bbb in term 2.
I20260812 19:56:48.202819 17194 leader_election.cc:304] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, de82d3b77b4449468e9cbe58c1180bbb; no voters:
I20260812 19:56:48.203250 17472 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:56:48.204500 17472 raft_consensus.cc:697] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 LEADER]: Becoming Leader. State: Replica: de82d3b77b4449468e9cbe58c1180bbb, State: Running, Role: LEADER
I20260812 19:56:48.204818 17424 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de82d3b77b4449468e9cbe58c1180bbb in term 2.
I20260812 19:56:48.205200 17472 consensus_queue.cc:237] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:48.208704 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader de82d3b77b4449468e9cbe58c1180bbb. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.209031 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:48.209822 17477 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:56:48.211943 17477 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:48.212873 17477 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:48.212998 17477 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:48.214385 17477 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:48.215341 17477 catalog_manager.cc:6103] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Loaded TSK: 0
I20260812 19:56:48.216281 17477 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:48.363134 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:48.588173 17424 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 2 FOLLOWER]: Refusing update from remote peer de82d3b77b4449468e9cbe58c1180bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:48.589339 17475 consensus_queue.cc:1048] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:48.593427 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de82d3b77b4449468e9cbe58c1180bbb. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.593746 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:48.595785 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.596118 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:48.597069 17320 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Refusing update from remote peer de82d3b77b4449468e9cbe58c1180bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:56:48.597249 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.597496 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:48.598058 17475 consensus_queue.cc:1048] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:56:48.602990 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader de82d3b77b4449468e9cbe58c1180bbb. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.602756 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:48.603307 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:48.603380 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:48.757270 17482 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:48.757705 17482 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:48.757839 17482 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:48.766566 17482 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:48.766883 17482 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:48.778685 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:48.780568 17482 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:48.782276 17482 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:48.793926 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:48.794353 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:48.794024 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:48.795035 17482 server_base.cc:1061] running on GCE node
I20260812 19:56:48.795744 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:48.797072 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:48.798460 17482 hybrid_clock.cc:648] HybridClock initialized: now 1786564608798297 us; error 123 us; skew 500 ppm
I20260812 19:56:48.798831 17482 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:48 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564608, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:48.803123 17482 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:48.805150 17482 webserver.cc:533] Webserver started at http://127.25.14.1:34361/ using document root <none> and password file <none>
I20260812 19:56:48.806054 17482 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:48.806166 17482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:48.812876 17482 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:56:48.816382 17509 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:48.818094 17482 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:56:48.818296 17482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:48.819123 17482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:48.837918 17482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:48.842190 17482 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:48.842514 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:48.843668 17482 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:48.845474 17482 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:48.845577 17482 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:48.845772 17482 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:48.845880 17482 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:48.884522 17633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41407 every 8 connection(s)
I20260812 19:56:48.884516 17482 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41407
I20260812 19:56:48.886672 17482 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:48.889478 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17482
I20260812 19:56:48.889780 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:56:48 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564608, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:48 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564608, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:56:48.949642 17635 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:48.950115 17635 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:48.951416 17635 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:48.954479 17211 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
Aug 12 19:56:48 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564608, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:48.957104 17211 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:45051
I20260812 19:56:48.964360 17634 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:48.964533 17634 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:48.965039 17634 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:48.967288 17413 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
I20260812 19:56:48.967545 17638 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:48.967707 17638 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:48.968016 17638 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:48.969960 17309 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:49.057596 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:49.104256 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "de82d3b77b4449468e9cbe58c1180bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:49.104545 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: LEADER
W20260812 19:56:49.439682 17651 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:49.440063 17651 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:49.440157 17651 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:49.449178 17651 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:49.449391 17651 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:56:49.460852 17651 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:49.462688 17651 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:49.464124 17651 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:49.475517 17663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:49.475546 17662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:49.476315 17665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:49.477077 17651 server_base.cc:1061] running on GCE node
I20260812 19:56:49.477880 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:49.479221 17651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:49.480475 17651 hybrid_clock.cc:648] HybridClock initialized: now 1786564609480347 us; error 91 us; skew 500 ppm
I20260812 19:56:49.480872 17651 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:49 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:49.484938 17651 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:56:49.486981 17651 webserver.cc:533] Webserver started at http://127.25.14.2:45417/ using document root <none> and password file <none>
I20260812 19:56:49.487891 17651 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:49.488054 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:49.494431 17651 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:49.497967 17672 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:49.499759 17651 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:49.499991 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "49396012b74d4831b8a388f967c9d707"
format_stamp: "Formatted at 2026-08-12 19:56:39 on dist-test-slave-p3t5"
I20260812 19:56:49.500839 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:49.534319 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:49.538908 17651 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:49.539247 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:49.540580 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:49.542917 17651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:49.543042 17651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:49.543155 17651 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:49.543236 17651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:49.585278 17651 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:45273
I20260812 19:56:49.585376 17807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:45273 every 8 connection(s)
I20260812 19:56:49.587292 17651 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:56:49.593578 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17651
I20260812 19:56:49.593837 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:56:49 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:49 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:49.628795 17809 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:49.629292 17809 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:49.630544 17809 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:49.631507 17808 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:49.631657 17808 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:49.632242 17808 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:49.633636 17211 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:45273)
Aug 12 19:56:49 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564609, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:49.634262 17413 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:45273)
I20260812 19:56:49.634812 17211 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:58805
I20260812 19:56:49.642020 17810 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:49.642230 17810 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:49.642657 17810 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:49.643939 17309 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:45273)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:56:49.757294 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:49.961472 17635 heartbeater.cc:499] Master 127.25.14.62:41611 was elected leader, sending a full tablet report...
W20260812 19:56:50.142068 17822 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:50.142479 17822 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:50.142637 17822 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:50.151400 17822 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:50.151683 17822 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:56:50.162755 17822 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:50.164616 17822 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:50.166180 17822 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:50.178676 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:50.179226 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:50.180713 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:50.181712 17822 server_base.cc:1061] running on GCE node
I20260812 19:56:50.182420 17822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:50.183679 17822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:50.187013 17822 hybrid_clock.cc:648] HybridClock initialized: now 1786564610186819 us; error 140 us; skew 500 ppm
I20260812 19:56:50.187501 17822 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:50 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:50.191722 17822 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:56:50.193780 17822 webserver.cc:533] Webserver started at http://127.25.14.3:45817/ using document root <none> and password file <none>
I20260812 19:56:50.194645 17822 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:50.194816 17822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:50.201078 17822 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 19:56:50.204573 17839 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:50.206180 17822 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:56:50.206382 17822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "b66bd38d1f31472bb23a900f36240af1"
format_stamp: "Formatted at 2026-08-12 19:56:40 on dist-test-slave-p3t5"
I20260812 19:56:50.207299 17822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:50.234822 17822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:50.239434 17822 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:50.239758 17822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:50.240921 17822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:50.242789 17822 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:50.242929 17822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:50.243072 17822 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:50.243167 17822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:50.280802 17822 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:44397
I20260812 19:56:50.280983 17969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:44397 every 8 connection(s)
I20260812 19:56:50.282604 17822 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:56:50.292434 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17822
I20260812 19:56:50.292706 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:56:50 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:56:50 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:56:50 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:50.313808 17971 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:50.313962 17970 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:50.314333 17970 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:50.314339 17971 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:50.314896 17973 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:50.315120 17973 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:50.315878 17971 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:50.315928 17973 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:50.316004 17970 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:50.317929 17211 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:44397)
I20260812 19:56:50.318164 17309 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:44397)
I20260812 19:56:50.318461 17412 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:44397)
I20260812 19:56:50.319139 17211 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:42369
I20260812 19:56:50.334080 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:50.343118 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:56:50.343315 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17175
I20260812 19:56:50.344372 17246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:50.460764 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17175
I20260812 19:56:50.491354 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--webserver_interface=127.25.14.62
--webserver_port=45195
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:50.637773 17809 heartbeater.cc:499] Master 127.25.14.62:41611 was elected leader, sending a full tablet report...
W20260812 19:56:50.639525 17809 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111)
W20260812 19:56:50.884220 17985 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:50.884603 17985 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:50.884769 17985 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:50.893909 17985 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:56:50.894032 17985 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:50.894114 17985 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:56:50.894196 17985 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:56:50.905798 17985 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=45195
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:50.907682 17985 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:50.909248 17985 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:50.918637 17991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:50.918713 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:50.918953 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:50.920058 17985 server_base.cc:1061] running on GCE node
I20260812 19:56:50.920943 17985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:50.922823 17985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:50.924173 17985 hybrid_clock.cc:648] HybridClock initialized: now 1786564610924014 us; error 115 us; skew 500 ppm
I20260812 19:56:50.924635 17985 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:50 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:50.928874 17985 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:56:50.930984 17985 webserver.cc:533] Webserver started at http://127.25.14.62:45195/ using document root <none> and password file <none>
I20260812 19:56:50.931821 17985 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:50.931983 17985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:50.938594 17985 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:56:50.942543 18001 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:50.944417 17985 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:50.944630 17985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "de82d3b77b4449468e9cbe58c1180bbb"
format_stamp: "Formatted at 2026-08-12 19:56:36 on dist-test-slave-p3t5"
I20260812 19:56:50.945690 17985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 19:56:50.966207 17635 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:41611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111)
I20260812 19:56:50.975925 17985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:50.980330 17985 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:50.980588 17985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:51.003762 17985 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41611
I20260812 19:56:51.003800 18065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41611 every 8 connection(s)
I20260812 19:56:51.005873 17985 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:56:51.006620 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17985
I20260812 19:56:51.007002 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17482
I20260812 19:56:51.015761 18066 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:56:51 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:56:51.020339 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap starting.
I20260812 19:56:51.028805 17629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:51.042488 18066 log.cc:826] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Log is configured to *not* fsync() on all Append() calls
I20260812 19:56:51.052819 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:56:51.053409 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: Bootstrap complete.
I20260812 19:56:51.060745 18066 raft_consensus.cc:359] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:51.061551 18066 raft_consensus.cc:740] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de82d3b77b4449468e9cbe58c1180bbb, State: Initialized, Role: FOLLOWER
I20260812 19:56:51.062415 18066 consensus_queue.cc:260] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:51.063416 18078 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:51.063746 18078 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:51.064692 18066 sys_catalog.cc:565] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:56:51.076413 18090 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:51.076653 18090 catalog_manager.cc:1588] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: loading cluster ID for follower catalog manager: success
I20260812 19:56:51.081266 18090 catalog_manager.cc:1610] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: acquiring CA information for follower catalog manager: success
I20260812 19:56:51.084275 18090 catalog_manager.cc:1638] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:56:51.127683 17985 master_runner.cc:264] No leader master found from master de82d3b77b4449468e9cbe58c1180bbb
I20260812 19:56:51.127837 17985 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:51.135332 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17482
I20260812 19:56:51.164328 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41407
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=34361
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:51.321848 17971 heartbeater.cc:499] Master 127.25.14.62:41611 was elected leader, sending a full tablet report...
I20260812 19:56:51.333926 18022 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b66bd38d1f31472bb23a900f36240af1" instance_seqno: 1786564610271688) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:53013; Asking this server to re-register.
I20260812 19:56:51.335227 17971 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:51.337406 18022 ts_manager.cc:194] Registered new tserver with Master: b66bd38d1f31472bb23a900f36240af1 (127.25.14.3:44397)
W20260812 19:56:51.583230 18097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:51.583604 18097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:51.583707 18097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:51.595402 18097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:51.595626 18097 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:51.598067 18104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader de82d3b77b4449468e9cbe58c1180bbb)
I20260812 19:56:51.598340 18104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:51.600682 18104 leader_election.cc:290] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611), b9c4be8171d04121a8c2aa7f27c50087 (127.25.14.60:44601)
I20260812 19:56:51.609385 18034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19784c935a35442f9d8195a4df4f38ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de82d3b77b4449468e9cbe58c1180bbb" is_pre_election: true
I20260812 19:56:51.610113 18034 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19784c935a35442f9d8195a4df4f38ce in term 2.
I20260812 19:56:51.610495 17424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19784c935a35442f9d8195a4df4f38ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9c4be8171d04121a8c2aa7f27c50087" is_pre_election: true
I20260812 19:56:51.610978 17295 leader_election.cc:304] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, de82d3b77b4449468e9cbe58c1180bbb; no voters:
I20260812 19:56:51.611465 18104 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:56:51.611624 18104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader de82d3b77b4449468e9cbe58c1180bbb)
I20260812 19:56:51.611768 18104 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:51.612401 18097 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34361
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:51.614482 18097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:51.614811 18104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:51.615563 18104 leader_election.cc:290] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [CANDIDATE]: Term 3 election: Requested vote from peers de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611), b9c4be8171d04121a8c2aa7f27c50087 (127.25.14.60:44601)
I20260812 19:56:51.616400 17424 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19784c935a35442f9d8195a4df4f38ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9c4be8171d04121a8c2aa7f27c50087"
I20260812 19:56:51.616462 18034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19784c935a35442f9d8195a4df4f38ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de82d3b77b4449468e9cbe58c1180bbb"
I20260812 19:56:51.616686 18097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 19:56:51.616730 18034 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:51.622889 18034 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19784c935a35442f9d8195a4df4f38ce in term 3.
I20260812 19:56:51.623587 17295 leader_election.cc:304] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19784c935a35442f9d8195a4df4f38ce, de82d3b77b4449468e9cbe58c1180bbb; no voters: b9c4be8171d04121a8c2aa7f27c50087
I20260812 19:56:51.624074 18104 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:56:51.624577 18104 raft_consensus.cc:697] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [term 3 LEADER]: Becoming Leader. State: Replica: 19784c935a35442f9d8195a4df4f38ce, State: Running, Role: LEADER
I20260812 19:56:51.625049 18104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
I20260812 19:56:51.626859 18107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19784c935a35442f9d8195a4df4f38ce. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:51.627102 18107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:51.630285 18111 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
W20260812 19:56:51.631690 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:51.632309 18110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:51.633127 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:51.633476 18097 server_base.cc:1061] running on GCE node
I20260812 19:56:51.633667 18111 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:56:51.634253 18097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:51.634615 18111 catalog_manager.cc:1295] Loaded cluster ID: 0e30eda779014c54bf70d2973b7064ca
I20260812 19:56:51.634719 18111 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:56:51.635669 18097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:51.636039 18111 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:56:51.637125 18111 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce: Loaded TSK: 0
I20260812 19:56:51.638118 18111 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:56:51.639093 18097 hybrid_clock.cc:648] HybridClock initialized: now 1786564611638858 us; error 175 us; skew 500 ppm
I20260812 19:56:51.639523 18097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:51 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564611, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:51.644398 18097 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:51.646697 18097 webserver.cc:533] Webserver started at http://127.25.14.1:34361/ using document root <none> and password file <none>
I20260812 19:56:51.647783 18097 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:51.647940 18097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:51.648327 17809 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:51.648561 17809 heartbeater.cc:499] Master 127.25.14.62:41611 was elected leader, sending a full tablet report...
I20260812 19:56:51.649593 18022 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "49396012b74d4831b8a388f967c9d707" instance_seqno: 1786564609576051) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:55299; Asking this server to re-register.
I20260812 19:56:51.650390 17809 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:51.651741 18022 ts_manager.cc:194] Registered new tserver with Master: 49396012b74d4831b8a388f967c9d707 (127.25.14.2:45273)
I20260812 19:56:51.654870 18097 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:56:51.657939 18123 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:51.659448 18097 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:56:51.659636 18097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:51.660490 18097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:51.679774 18097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:51.683877 18097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:51.684252 18097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:51.685328 18097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:51.687227 18097 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:51.687323 18097 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:51.687412 18097 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:51.687469 18097 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:51.724076 18097 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41407
I20260812 19:56:51.724188 18249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41407 every 8 connection(s)
I20260812 19:56:51.725881 18097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:51.731710 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18097
Aug 12 19:56:51 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564611, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:56:51.750948 18262 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader de82d3b77b4449468e9cbe58c1180bbb)
Aug 12 19:56:51 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564611, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:56:51.751221 18262 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } }
Aug 12 19:56:51 dist-test-slave-p3t5 krb5kdc[17150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564611, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:56:51.762588 18251 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41611
I20260812 19:56:51.763049 18251 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:51.764264 18251 heartbeater.cc:507] Master 127.25.14.62:41611 requested a full tablet report, sending...
I20260812 19:56:51.764295 17320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "19784c935a35442f9d8195a4df4f38ce" is_pre_election: true
I20260812 19:56:51.765941 18262 leader_election.cc:290] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de82d3b77b4449468e9cbe58c1180bbb (127.25.14.62:41611), 19784c935a35442f9d8195a4df4f38ce (127.25.14.61:35133)
I20260812 19:56:51.766772 18022 ts_manager.cc:194] Registered new tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
I20260812 19:56:51.772239 18254 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35133
I20260812 19:56:51.772440 18254 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:51.773082 18254 heartbeater.cc:507] Master 127.25.14.61:35133 requested a full tablet report, sending...
I20260812 19:56:51.773500 18250 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44601
I20260812 19:56:51.773730 18250 heartbeater.cc:461] Registering TS with master...
I20260812 19:56:51.774242 18250 heartbeater.cc:507] Master 127.25.14.60:44601 requested a full tablet report, sending...
I20260812 19:56:51.774613 17309 ts_manager.cc:194] Re-registered known tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
I20260812 19:56:51.776525 17309 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:52305
I20260812 19:56:51.776966 18034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9c4be8171d04121a8c2aa7f27c50087" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de82d3b77b4449468e9cbe58c1180bbb" is_pre_election: true
I20260812 19:56:51.777429 18034 raft_consensus.cc:2393] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9c4be8171d04121a8c2aa7f27c50087 in current term 3: Already voted for candidate 19784c935a35442f9d8195a4df4f38ce in this term.
I20260812 19:56:51.777691 17412 ts_manager.cc:194] Re-registered known tserver with Master: 3cc1f2231eec4252be01dd58bc361f9d (127.25.14.1:41407)
I20260812 19:56:51.778820 17398 leader_election.cc:304] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9c4be8171d04121a8c2aa7f27c50087; no voters: 19784c935a35442f9d8195a4df4f38ce, de82d3b77b4449468e9cbe58c1180bbb
I20260812 19:56:51.779335 18262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:56:51.782088 18262 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:56:51.805704 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18097
W20260812 19:56:51.807273 18268 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:51.803076 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.25.14.1:41407)
0812 19:56:51.803952 (+ 876us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:51.803960 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:51.803994 (+ 34us) server_negotiation.cc:408] Connection header received
0812 19:56:51.804064 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:51.804071 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:51.804331 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:51.804532 (+ 201us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:51.805437 (+ 905us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59040: BlockingRecv error: recv got EOF from 127.0.0.1:59040 (error 108)
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":506,"threads_started":1}
W20260812 19:56:51.808529 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18097: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:41407: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:51.808650 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 18097
I20260812 19:56:51.834172 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41407
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=34361
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:52.216364 18034 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [term 3 FOLLOWER]: Refusing update from remote peer 19784c935a35442f9d8195a4df4f38ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:52.218057 18271 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:52.222970 18272 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19784c935a35442f9d8195a4df4f38ce. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.223286 18272 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:52.226135 18274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.226441 18274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:52.227156 17424 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [term 3 FOLLOWER]: Refusing update from remote peer 19784c935a35442f9d8195a4df4f38ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:56:52.230517 18272 sys_catalog.cc:455] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.230739 18272 sys_catalog.cc:458] T 00000000000000000000000000000000 P de82d3b77b4449468e9cbe58c1180bbb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:52.234082 18274 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:56:52.240146 18262 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19784c935a35442f9d8195a4df4f38ce. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.240094 18282 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.240437 18262 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:56:52.240435 18282 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c4be8171d04121a8c2aa7f27c50087 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:56:52.252876 18270 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:52.253235 18270 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:52.253340 18270 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:56:52.253397 18270 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:52.262221 18270 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:52.262457 18270 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:52.274304 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34361
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:52.276036 18270 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:52.277457 18270 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:52.288594 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:52.289108 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:52.288524 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:52.289873 18270 server_base.cc:1061] running on GCE node
I20260812 19:56:52.290643 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:52.291896 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:52.293231 18270 hybrid_clock.cc:648] HybridClock initialized: now 1786564612293056 us; error 123 us; skew 500 ppm
I20260812 19:56:52.293597 18270 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:52 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564612, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:52.297601 18270 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:52.299569 18270 webserver.cc:533] Webserver started at http://127.25.14.1:34361/ using document root <none> and password file <none>
I20260812 19:56:52.300360 18270 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:52.300448 18270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:52.306613 18270 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:56:52.310036 18298 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:52.311605 18270 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:56:52.311800 18270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:52.312565 18270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:52.335301 18270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:52.339682 18270 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:52.340087 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:52.341393 18270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:52.343859 18270 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:52.343971 18270 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:52.344165 18270 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:52.344245 18270 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:56:52.344694 18270 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:56:52.344820 18270 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:56:52.344925 18270 fault_injection.cc:59] injecting fault for kudu (pid 18270): FLAGS_fault_before_start (process will exit)
I20260812 19:56:52.370232 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41407
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--webserver_port=34361
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:56:52.649828 17810 heartbeater.cc:499] Master 127.25.14.61:35133 was elected leader, sending a full tablet report...
W20260812 19:56:52.756596 18306 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:56:52.757058 18306 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:56:52.757228 18306 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:56:52.766377 18306 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:56:52.766664 18306 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:56:52.778723 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42225,127.25.14.21:33609,127.25.14.22:44435,127.25.14.23:43175,127.25.14.24:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34361
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:41611,127.25.14.61:35133,127.25.14.60:44601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:56:52.780613 18306 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:56:52.782297 18306 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:56:52.794730 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:52.794852 18313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:56:52.794920 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:56:52.796423 18306 server_base.cc:1061] running on GCE node
I20260812 19:56:52.797187 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:56:52.798561 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:56:52.799983 18306 hybrid_clock.cc:648] HybridClock initialized: now 1786564612799811 us; error 132 us; skew 500 ppm
I20260812 19:56:52.800419 18306 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:56:52 dist-test-slave-p3t5 krb5kdc[17150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564612, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:56:52.804965 18306 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:56:52.807173 18306 webserver.cc:533] Webserver started at http://127.25.14.1:34361/ using document root <none> and password file <none>
I20260812 19:56:52.808018 18306 fs_manager.cc:362] Metadata directory not provided
I20260812 19:56:52.808169 18306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:56:52.815011 18306 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:52.818380 18324 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:52.820113 18306 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:56:52.820338 18306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3cc1f2231eec4252be01dd58bc361f9d"
format_stamp: "Formatted at 2026-08-12 19:56:38 on dist-test-slave-p3t5"
I20260812 19:56:52.821203 18306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:56:52.841779 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "19784c935a35442f9d8195a4df4f38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de82d3b77b4449468e9cbe58c1180bbb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41611 } } peers { permanent_uuid: "19784c935a35442f9d8195a4df4f38ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35133 } } peers { permanent_uuid: "b9c4be8171d04121a8c2aa7f27c50087" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44601 } } }
I20260812 19:56:52.842141 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19784c935a35442f9d8195a4df4f38ce [sys.catalog]: This master's current role is: LEADER
I20260812 19:56:52.858713 18306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:56:52.863127 18306 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:56:52.863394 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:56:52.864395 18306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:56:52.866349 18306 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:56:52.866452 18306 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:52.866534 18306 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:56:52.866591 18306 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:56:52.904560 18306 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41407
I20260812 19:56:52.904623 18453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41407 every 8 connection(s)
I20260812 19:56:52.906370 18306 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:56:52.916477 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18306
I20260812 19:56:52.916913 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18306
W20260812 19:56:52.933465 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18306: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:41407: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:52.933704 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 18306
I20260812 19:56:52.958225 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17651
W20260812 19:56:52.959103 18464 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:52.912485 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:47479 (local address 127.25.14.60:44601)
0812 19:56:52.930913 (+ 18428us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:52.930918 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:52.930944 (+ 26us) server_negotiation.cc:408] Connection header received
0812 19:56:52.931001 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:52.931005 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:52.931070 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:52.931554 (+ 484us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:52.933551 (+ 1997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:52.937194 (+ 3643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:52.957992 (+ 20798us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:47479: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":18317,"thread_start_us":18152,"threads_started":1}
W20260812 19:56:52.959122 18465 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:52.913429 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58327 (local address 127.25.14.61:35133)
0812 19:56:52.930877 (+ 17448us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:52.930882 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:52.930902 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:56:52.930968 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:52.930973 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:52.931041 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:52.931191 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:52.945837 (+ 14646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:52.947391 (+ 1554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:52.958036 (+ 10645us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58327: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17306,"thread_start_us":17079,"threads_started":1}
W20260812 19:56:52.959123 18461 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:52.910326 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:51895 (local address 127.25.14.62:41611)
0812 19:56:52.910820 (+ 494us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:52.910824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:52.911778 (+ 954us) server_negotiation.cc:408] Connection header received
0812 19:56:52.915377 (+ 3599us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:52.915393 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:52.915534 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:52.915681 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:52.919897 (+ 4216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:52.922393 (+ 2496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:56:52.925410 (+ 3017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:56:52.926074 (+ 664us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:56:52.958164 (+ 32090us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:51895: BlockingRecv error: recv got EOF from 127.25.14.1:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
W20260812 19:56:52.961764 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17651: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:45273: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:52.961966 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17651
I20260812 19:56:52.990136 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17822
W20260812 19:56:52.993188 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17822: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:44397: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:56:52.993384 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17822
I20260812 19:56:53.017918 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17985
W20260812 19:56:53.020527 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17985: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:41611: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:56:53.020666 18461 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:56:53.019075 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.25.14.62:41611)
0812 19:56:53.019299 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0812 19:56:53.019307 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:56:53.019331 (+ 24us) server_negotiation.cc:408] Connection header received
0812 19:56:53.019413 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:56:53.019418 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:56:53.019494 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:56:53.019613 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:56:53.020381 (+ 768us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32938: BlockingRecv error: recv got EOF from 127.0.0.1:32938 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 19:56:53.020730 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17985
I20260812 19:56:53.050078 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17275
I20260812 19:56:53.051486 17340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:53.066689 17375 master_runner.cc:187] Error getting master registration for 127.25.14.62:41611: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41611: connect: Connection refused (error 111),
I20260812 19:56:53.070314 17375 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:56:53.167469 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17275
I20260812 19:56:53.197824 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17375
I20260812 19:56:53.199162 17444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:56:53.313234 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17375
2026-08-12T19:56:53Z chronyd exiting
2026-08-12T19:56:53Z chronyd exiting
2026-08-12T19:56:53Z chronyd exiting
2026-08-12T19:56:53Z chronyd exiting
2026-08-12T19:56:53Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/8 (20226 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/9
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:56:53 dist-test-slave-p3t5 krb5kdc[18477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:56:53 dist-test-slave-p3t5 krb5kdc[18477](info): set up 2 sockets
Aug 12 19:56:53 dist-test-slave-p3t5 krb5kdc[18477](info): commencing operation
krb5kdc: starting...
W20260812 19:56:55.592885 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.183s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:56:55 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:57:03: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:57:06 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564626, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:57:12 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564632, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:57:12 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564632, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:12.626959 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:13.009750 18810 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:13.010361 18810 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:13.010560 18810 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:13.021113 18810 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:13.021247 18810 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:13.021351 18810 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:13.021431 18810 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:13.034063 18810 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:13.035900 18810 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:13.037412 18810 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:13.047477 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:13.048026 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:13.047345 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:13.048820 18810 server_base.cc:1061] running on GCE node
I20260812 19:57:13.049696 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:13.049876 18810 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:13.049950 18810 hybrid_clock.cc:648] HybridClock initialized: now 1786564633049930 us; error 0 us; skew 500 ppm
I20260812 19:57:13.050271 18810 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:13.054072 18810 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:57:13.056348 18810 webserver.cc:533] Webserver started at http://127.25.14.62:45217/ using document root <none> and password file <none>
I20260812 19:57:13.057179 18810 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:13.057349 18810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:13.057827 18810 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:13.060417 18810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.061291 18810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.067203 18810 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 19:57:13.071156 18827 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:13.072728 18810 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:57:13.072952 18810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.073184 18810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:13.108915 18810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:13.113148 18810 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:13.113490 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:13.136101 18810 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43693
I20260812 19:57:13.136125 18880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43693 every 8 connection(s)
I20260812 19:57:13.137939 18810 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:57:13.142709 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:57:13.143206 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18810
I20260812 19:57:13.143429 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:57:13.149006 18881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
I20260812 19:57:13.175595 18881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:57:13.177031 18828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
W20260812 19:57:13.178892 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:13.228305 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } attempt: 1
W20260812 19:57:13.230727 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:13.286444 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40793
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:13.299135 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } attempt: 2
W20260812 19:57:13.301535 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
I20260812 19:57:13.392939 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } attempt: 3
W20260812 19:57:13.395502 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
I20260812 19:57:13.538851 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } attempt: 4
W20260812 19:57:13.541570 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111)
W20260812 19:57:13.674533 18891 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:13.674913 18891 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:13.675032 18891 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:13.683561 18891 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:13.683688 18891 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:13.683751 18891 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:13.683794 18891 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:13.695159 18891 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:13.696936 18891 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:13.698410 18891 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:13.707335 18898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:13.707322 18899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:13.708249 18901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:13.708568 18891 server_base.cc:1061] running on GCE node
I20260812 19:57:13.709457 18891 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:13.709671 18891 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:13.709784 18891 hybrid_clock.cc:648] HybridClock initialized: now 1786564633709754 us; error 0 us; skew 500 ppm
I20260812 19:57:13.710110 18891 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:13.713974 18891 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:57:13.716281 18891 webserver.cc:533] Webserver started at http://127.25.14.61:39893/ using document root <none> and password file <none>
I20260812 19:57:13.717127 18891 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:13.717281 18891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:13.717761 18891 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:13.720436 18891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "63916aa6f5da40a4bdf497abe2f04657"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.721347 18891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "63916aa6f5da40a4bdf497abe2f04657"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.727563 18891 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 19:57:13.731710 18908 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:13.733892 18891 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:57:13.734112 18891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "63916aa6f5da40a4bdf497abe2f04657"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:13.734323 18891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:13.755168 18891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:13.759784 18891 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:13.760133 18891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:13.783838 18891 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:40793
I20260812 19:57:13.783861 18962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:40793 every 8 connection(s)
I20260812 19:57:13.785729 18891 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:57:13.791215 18963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:57:13.793712 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18891
I20260812 19:57:13.794003 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:57:13.797878 18963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:13.822000 18963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:13.839357 18963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:57:13.840854 18910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:13.840898 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } attempt: 5
Aug 12 19:57:13 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
W20260812 19:57:13.849043 18963 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:13.856885 18881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:57:13.859261 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:13.898460 18963 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 1
W20260812 19:57:13.900877 18963 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 19:57:13.910477 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 1
W20260812 19:57:13.912726 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:13.949066 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45943
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:13.969118 18963 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 2
W20260812 19:57:13.972091 18963 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:13.980993 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 2
W20260812 19:57:13.983280 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:14.063403 18963 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 3
W20260812 19:57:14.066015 18963 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:14.089586 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 3
W20260812 19:57:14.091970 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:14.209340 18963 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 4
W20260812 19:57:14.212138 18963 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
I20260812 19:57:14.269281 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 4
W20260812 19:57:14.271900 18881 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111)
W20260812 19:57:14.336338 18977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:14.336694 18977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:14.336927 18977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:14.345726 18977 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:14.345849 18977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:14.345995 18977 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:14.346096 18977 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:14.357637 18977 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:14.359526 18977 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:14.361105 18977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:14.370848 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:14.370772 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:14.370888 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:14.371959 18977 server_base.cc:1061] running on GCE node
I20260812 19:57:14.372969 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:14.373174 18977 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:14.373297 18977 hybrid_clock.cc:648] HybridClock initialized: now 1786564634373255 us; error 0 us; skew 500 ppm
I20260812 19:57:14.373749 18977 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564634, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:14.377619 18977 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:57:14.380121 18977 webserver.cc:533] Webserver started at http://127.25.14.60:35723/ using document root <none> and password file <none>
I20260812 19:57:14.380949 18977 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:14.381090 18977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:14.381510 18977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:14.384054 18977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
format_stamp: "Formatted at 2026-08-12 19:57:14 on dist-test-slave-p3t5"
I20260812 19:57:14.384894 18977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
format_stamp: "Formatted at 2026-08-12 19:57:14 on dist-test-slave-p3t5"
I20260812 19:57:14.390690 18977 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 19:57:14.394409 18992 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:14.396066 18977 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:57:14.396260 18977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
format_stamp: "Formatted at 2026-08-12 19:57:14 on dist-test-slave-p3t5"
I20260812 19:57:14.396483 18977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:14.435846 18977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:14.440546 18977 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:14.440872 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:14.462915 18977 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:45943
I20260812 19:57:14.462963 19049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:45943 every 8 connection(s)
I20260812 19:57:14.464655 18977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:57:14.466989 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18977
I20260812 19:57:14.467219 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:57:14.470501 19050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:57:14.480490 19050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564634, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:14.505190 19050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:57:14.511391 18963 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 5
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564634, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:14.522771 19050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564634, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:14.537012 18963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Bootstrap starting.
I20260812 19:57:14.541298 18963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Neither blocks nor log segments found. Creating new log.
I20260812 19:57:14.542580 18963 log.cc:826] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:14.544888 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Bootstrap starting.
I20260812 19:57:14.545902 18963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: No bootstrap required, opened a new log
I20260812 19:57:14.548889 19050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Neither blocks nor log segments found. Creating new log.
I20260812 19:57:14.549190 18881 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } attempt: 5
I20260812 19:57:14.550379 19050 log.cc:826] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:14.553090 18963 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.553665 18963 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:57:14.553931 18963 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63916aa6f5da40a4bdf497abe2f04657, State: Initialized, Role: FOLLOWER
I20260812 19:57:14.554080 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: No bootstrap required, opened a new log
I20260812 19:57:14.554841 18963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.556890 19064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.557237 19064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 19:57:14 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:14.563760 19050 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.564308 19050 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:57:14.564517 19050 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d441cbaf4244a3d98862fb6f936f0bf, State: Initialized, Role: FOLLOWER
I20260812 19:57:14.565003 18963 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:14.565559 19050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.569962 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.570317 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.571651 19050 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:14.580165 18881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap starting.
I20260812 19:57:14.586143 18881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Neither blocks nor log segments found. Creating new log.
I20260812 19:57:14.588099 18881 log.cc:826] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:14.598218 18881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: No bootstrap required, opened a new log
I20260812 19:57:14.591639 18891 master_runner.cc:187] Error getting master registration for 127.25.14.62:43693: OK, instance_id { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" instance_seqno: 1786564633130049 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:57:14.606856 18891 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:57:14.608036 19082 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:57:14.608181 19082 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:57:14.608951 18881 raft_consensus.cc:359] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.609562 18881 raft_consensus.cc:385] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:57:14.609831 18881 raft_consensus.cc:740] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede2f0430904328b231f15c3425a9cf, State: Initialized, Role: FOLLOWER
I20260812 19:57:14.600582 18977 master_runner.cc:187] Error getting master registration for 127.25.14.62:43693: OK, instance_id { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" instance_seqno: 1786564633130049 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:57:14.610423 18977 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:57:14.611164 18881 consensus_queue.cc:260] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.612915 19064 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:57:14.613206 19064 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
W20260812 19:57:14.614984 19091 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:57:14.615186 19091 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:57:14.615304 19064 leader_election.cc:290] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693), 3d441cbaf4244a3d98862fb6f936f0bf (127.25.14.60:45943)
I20260812 19:57:14.616036 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.616529 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.616596 19023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" is_pre_election: true
I20260812 19:57:14.617235 19023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63916aa6f5da40a4bdf497abe2f04657 in term 0.
I20260812 19:57:14.618503 18910 leader_election.cc:304] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d441cbaf4244a3d98862fb6f936f0bf, 63916aa6f5da40a4bdf497abe2f04657; no voters:
I20260812 19:57:14.618980 19064 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:57:14.619165 19064 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:57:14.619290 19064 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:57:14.620980 18881 sys_catalog.cc:565] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:14.621057 18854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede2f0430904328b231f15c3425a9cf" is_pre_election: true
I20260812 19:57:14.621665 18854 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63916aa6f5da40a4bdf497abe2f04657 in term 0.
I20260812 19:57:14.622912 19064 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.623742 19064 leader_election.cc:290] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 1 election: Requested vote from peers fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693), 3d441cbaf4244a3d98862fb6f936f0bf (127.25.14.60:45943)
I20260812 19:57:14.624241 18854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede2f0430904328b231f15c3425a9cf"
I20260812 19:57:14.624367 19023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
I20260812 19:57:14.624473 18854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:57:14.624637 19023 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:57:14.627990 18854 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63916aa6f5da40a4bdf497abe2f04657 in term 1.
I20260812 19:57:14.628383 19023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63916aa6f5da40a4bdf497abe2f04657 in term 1.
I20260812 19:57:14.628687 18911 leader_election.cc:304] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63916aa6f5da40a4bdf497abe2f04657, fede2f0430904328b231f15c3425a9cf; no voters:
I20260812 19:57:14.629340 19064 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:57:14.637081 19064 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 LEADER]: Becoming Leader. State: Replica: 63916aa6f5da40a4bdf497abe2f04657, State: Running, Role: LEADER
I20260812 19:57:14.637895 19064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:14.649863 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63916aa6f5da40a4bdf497abe2f04657. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.650305 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:14.668708 19103 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:57:14.671361 19103 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:57:14.687245 19112 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 19:57:14.688498 19023 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Refusing update from remote peer 63916aa6f5da40a4bdf497abe2f04657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:57:14.690358 18854 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Refusing update from remote peer 63916aa6f5da40a4bdf497abe2f04657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:57:14.691358 19064 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:57:14.692137 19064 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 19:57:14.693285 18999 tablet.cc:2406] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:57:14.719357 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63916aa6f5da40a4bdf497abe2f04657. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.719693 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.728276 19111 mvcc.cc:204] Tried to move back new op lower bound from 7317768743669694464 to 7317768743484170240. Current Snapshot: MvccSnapshot[applied={T|T < 7317768743669694464}]
I20260812 19:57:14.738312 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.738592 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.740011 19114 mvcc.cc:204] Tried to move back new op lower bound from 7317768743669694464 to 7317768743484170240. Current Snapshot: MvccSnapshot[applied={T|T < 7317768743669694464}]
I20260812 19:57:14.750437 19064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.750774 19064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:14.751889 19103 catalog_manager.cc:1383] Generated new cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:14.752022 19103 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:57:14.761718 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63916aa6f5da40a4bdf497abe2f04657. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.762013 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.764158 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.764424 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:14.782109 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:14.782392 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:14.788822 19103 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:57:14.790736 19103 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:57:14.825567 19103 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Generated new TSK 0
I20260812 19:57:14.826996 19103 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:14.982577 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:15.396625 19126 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:15.397020 19126 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:15.397125 19126 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:15.406937 19126 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:15.407157 19126 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:15.418113 19126 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:15.419894 19126 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:15.421298 19126 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:15.432044 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:15.432004 19134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:15.432392 19135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:15.433492 19126 server_base.cc:1061] running on GCE node
I20260812 19:57:15.434207 19126 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:15.434378 19126 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:15.434449 19126 hybrid_clock.cc:648] HybridClock initialized: now 1786564635434430 us; error 0 us; skew 500 ppm
I20260812 19:57:15.434724 19126 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:15 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564635, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:15.438174 19126 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:15.439932 19126 webserver.cc:533] Webserver started at http://127.25.14.1:40855/ using document root <none> and password file <none>
I20260812 19:57:15.440704 19126 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:15.440804 19126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:15.441133 19126 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:15.443818 19126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:15.444626 19126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:15.450672 19126 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260812 19:57:15.454627 19145 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:15.456187 19126 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:57:15.456358 19126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:15.456535 19126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:15.472990 19126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:15.478286 19126 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:15.478623 19126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:15.479799 19126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:15.482012 19126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:15.482112 19126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:15.482231 19126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:15.482297 19126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:15.519037 19126 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37683
I20260812 19:57:15.519086 19267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37683 every 8 connection(s)
I20260812 19:57:15.520622 19126 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:15.524052 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19126
I20260812 19:57:15.524276 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:57:15 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564635, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:15 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564635, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:57:15 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564635, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:15.572151 19270 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:15.572602 19270 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:15.574057 19270 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:15.577137 18926 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
I20260812 19:57:15.577525 19269 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:15.577731 19269 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:15.578284 19269 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:15.579221 18926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:35337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
I20260812 19:57:15.580673 18844 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
I20260812 19:57:15.583695 19268 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:15.583856 19268 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:15.584203 19268 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:15.585997 19012 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
I20260812 19:57:15.617702 19091 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:15.617838 19091 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: loading cluster ID for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:15.621012 19091 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: acquiring CA information for follower catalog manager: success
I20260812 19:57:15.623322 19091 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:15.692315 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:15.693912 19112 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:15.694053 19112 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: loading cluster ID for follower catalog manager: success
I20260812 19:57:15.698473 19112 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: acquiring CA information for follower catalog manager: success
I20260812 19:57:15.701467 19112 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:57:16.135551 19285 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:16.136049 19285 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:16.136212 19285 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:16.150475 19285 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:16.150770 19285 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:57:16.162398 19285 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:16.164147 19285 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:16.165560 19285 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:16.177131 19293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:16.177277 19295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:16.177228 19292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:16.178654 19285 server_base.cc:1061] running on GCE node
I20260812 19:57:16.179479 19285 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:16.179684 19285 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:16.179806 19285 hybrid_clock.cc:648] HybridClock initialized: now 1786564636179754 us; error 0 us; skew 500 ppm
I20260812 19:57:16.180226 19285 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:16 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:16.184233 19285 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:57:16.186223 19285 webserver.cc:533] Webserver started at http://127.25.14.2:43111/ using document root <none> and password file <none>
I20260812 19:57:16.187101 19285 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:16.187240 19285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:16.187651 19285 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:16.190593 19285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "2db9537736fa4ba289b33efdfb400a74"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.191493 19285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "2db9537736fa4ba289b33efdfb400a74"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.197953 19285 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 19:57:16.202010 19301 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:16.203646 19285 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:16.203866 19285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "2db9537736fa4ba289b33efdfb400a74"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.204105 19285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:16.220839 19285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:16.225750 19285 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:16.226028 19285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:16.227026 19285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:16.228812 19285 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:16.228890 19285 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:16.228982 19285 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:16.229019 19285 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:16.266104 19285 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:44205
I20260812 19:57:16.266222 19423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:44205 every 8 connection(s)
I20260812 19:57:16.267823 19285 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:57:16.273499 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19285
I20260812 19:57:16.273846 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:57:16 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:57:16 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:16 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:16.342379 19426 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:16.342468 19424 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:16.342434 19427 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:16.343000 19427 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:16.342993 19424 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:16.342994 19426 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:16.344156 19426 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:16.344187 19427 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:16.345726 19424 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:16.346176 18926 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:44205)
I20260812 19:57:16.346616 18844 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:44205)
I20260812 19:57:16.347167 19012 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:44205)
I20260812 19:57:16.347467 18926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:40741
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:16.479146 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:16.582787 19270 heartbeater.cc:499] Master 127.25.14.61:40793 was elected leader, sending a full tablet report...
W20260812 19:57:16.883563 19441 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:16.883937 19441 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:16.884152 19441 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:16.893163 19441 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:16.893460 19441 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:57:16.904995 19441 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:16.906914 19441 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:16.908420 19441 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:16.919413 19448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:16.919423 19447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:16.920006 19450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:16.920524 19441 server_base.cc:1061] running on GCE node
I20260812 19:57:16.921197 19441 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:16.921422 19441 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:16.921526 19441 hybrid_clock.cc:648] HybridClock initialized: now 1786564636921483 us; error 0 us; skew 500 ppm
I20260812 19:57:16.921902 19441 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:16 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:16.925698 19441 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:57:16.927489 19441 webserver.cc:533] Webserver started at http://127.25.14.3:40813/ using document root <none> and password file <none>
I20260812 19:57:16.928263 19441 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:16.928378 19441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:16.928694 19441 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:57:16.931416 19441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "e62b5dc6447f458f9135f070b3e771a4"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.932245 19441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "e62b5dc6447f458f9135f070b3e771a4"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.938439 19441 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 19:57:16.942435 19457 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:16.944095 19441 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:57:16.944345 19441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "e62b5dc6447f458f9135f070b3e771a4"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:16.944597 19441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:16.988093 19441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:16.992694 19441 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:16.993041 19441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:16.994282 19441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:16.996327 19441 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:16.996448 19441 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:16.996629 19441 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:16.996711 19441 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:17.035985 19441 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:38079
I20260812 19:57:17.036082 19582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:38079 every 8 connection(s)
I20260812 19:57:17.037588 19441 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:57:17.047438 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19441
I20260812 19:57:17.047729 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:17.087462 19584 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:17.087667 19583 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:17.087859 19585 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:17.088047 19584 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:17.088049 19585 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:17.088045 19583 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:17.089438 19585 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:17.089479 19584 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:17.089514 19583 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:17.091516 18926 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:38079)
I20260812 19:57:17.091671 19012 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:38079)
I20260812 19:57:17.091868 18844 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:38079)
I20260812 19:57:17.092690 18926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:52479
I20260812 19:57:17.098829 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:57:17.107606 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:57:17.107882 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18810
I20260812 19:57:17.108937 18876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:17.220743 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 18810
I20260812 19:57:17.252741 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--webserver_interface=127.25.14.62
--webserver_port=45217
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:17.349998 19427 heartbeater.cc:499] Master 127.25.14.61:40793 was elected leader, sending a full tablet report...
W20260812 19:57:17.350706 19426 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:43693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111)
W20260812 19:57:17.571199 18911 consensus_peers.cc:597] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 -> Peer fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693): Couldn't send request to peer fede2f0430904328b231f15c3425a9cf. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:57:17.587436 19269 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:43693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111)
W20260812 19:57:17.681592 19594 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:17.682089 19594 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:17.682252 19594 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:17.691439 19594 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:17.691602 19594 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:17.691726 19594 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:17.691818 19594 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:17.703682 19594 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=45217
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:17.705482 19594 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:17.707000 19594 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:17.716619 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:17.717484 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:17.718914 19611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:17.719158 19594 server_base.cc:1061] running on GCE node
I20260812 19:57:17.720199 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:17.720408 19594 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:17.720522 19594 hybrid_clock.cc:648] HybridClock initialized: now 1786564637720484 us; error 0 us; skew 500 ppm
I20260812 19:57:17.720906 19594 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:17.725188 19594 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:57:17.727813 19594 webserver.cc:533] Webserver started at http://127.25.14.62:45217/ using document root <none> and password file <none>
I20260812 19:57:17.728731 19594 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:17.728905 19594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:17.735380 19594 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:17.738814 19618 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:17.740608 19594 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:17.740825 19594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:17.741616 19594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:17.763598 19594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:17.772349 19594 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:17.772677 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:17.800802 19594 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43693
I20260812 19:57:17.800776 19677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43693 every 8 connection(s)
I20260812 19:57:17.802996 19594 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:57:17.812024 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19594
I20260812 19:57:17.812388 19678 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:57:17.812465 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19126
I20260812 19:57:17.819285 19678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap starting.
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:57:17.843079 19678 log.cc:826] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:17.853220 19263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:17.856933 19678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:57:17.857683 19678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap complete.
I20260812 19:57:17.865363 19678 raft_consensus.cc:359] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:17.866391 19678 raft_consensus.cc:740] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede2f0430904328b231f15c3425a9cf, State: Initialized, Role: FOLLOWER
I20260812 19:57:17.867466 19678 consensus_queue.cc:260] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:17.868557 19684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:17.869019 19684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:17.870012 19678 sys_catalog.cc:565] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:17.886584 19697 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:17.886860 19697 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: loading cluster ID for follower catalog manager: success
I20260812 19:57:17.901175 19697 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: acquiring CA information for follower catalog manager: success
I20260812 19:57:17.904656 19697 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:57:17 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:18.006738 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19126
I20260812 19:57:18.034607 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37683
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=40855
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:18.044716 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:18.045176 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:18.059311 19684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63916aa6f5da40a4bdf497abe2f04657. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:18.059651 19684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:18.095609 19585 heartbeater.cc:499] Master 127.25.14.61:40793 was elected leader, sending a full tablet report...
I20260812 19:57:18.107610 19640 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e62b5dc6447f458f9135f070b3e771a4" instance_seqno: 1786564637026509) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:53967; Asking this server to re-register.
I20260812 19:57:18.108669 19584 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:18.110872 19640 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:38079)
I20260812 19:57:18.363826 19426 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:18.365314 19640 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2db9537736fa4ba289b33efdfb400a74" instance_seqno: 1786564636256807) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:49849; Asking this server to re-register.
I20260812 19:57:18.366161 19426 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:18.367712 19640 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:44205)
W20260812 19:57:18.448042 19706 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:18.448396 19706 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:18.448493 19706 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:18.457513 19706 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:18.457809 19706 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:18.473467 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40855
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:18.475705 19706 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:18.477550 19706 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:18.489876 19714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:18.490528 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:18.490989 19718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:18.492513 19706 server_base.cc:1061] running on GCE node
I20260812 19:57:18.493238 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:18.493471 19706 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:18.493600 19706 hybrid_clock.cc:648] HybridClock initialized: now 1786564638493571 us; error 0 us; skew 500 ppm
I20260812 19:57:18.494016 19706 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:18 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564638, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:18.498095 19706 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:18.500100 19706 webserver.cc:533] Webserver started at http://127.25.14.1:40855/ using document root <none> and password file <none>
I20260812 19:57:18.501031 19706 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:18.501200 19706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:18.507571 19706 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:57:18.511044 19724 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:18.512651 19706 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:57:18.512867 19706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:18.513756 19706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:18.542696 19706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:18.547845 19706 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:18.548162 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:18.549316 19706 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:18.551455 19706 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:18.551599 19706 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:18.551751 19706 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:18.551851 19706 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:18.578034 19704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63916aa6f5da40a4bdf497abe2f04657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:18.578269 19704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:18.600013 19706 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37683
I20260812 19:57:18.600114 19852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37683 every 8 connection(s)
I20260812 19:57:18.601969 19706 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:18.610029 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19706
Aug 12 19:57:18 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 19:57:18 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564638, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:18 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564638, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:18.636953 19854 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:18.637447 19854 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:18.638773 19854 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:18.640672 19853 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:18.640847 19853 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:18.641090 19640 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
I20260812 19:57:18.641476 19853 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:18.643077 19012 ts_manager.cc:194] Re-registered known tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
Aug 12 19:57:18 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564638, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:18.655764 19855 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:18.656002 19855 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:18.656500 19855 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:18.658088 18926 ts_manager.cc:194] Re-registered known tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:37683)
I20260812 19:57:18.659299 18926 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:40453
I20260812 19:57:18.813231 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19706
W20260812 19:57:18.814706 19870 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:18.786199 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.25.14.1:37683)
0812 19:57:18.811123 (+ 24924us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:18.811130 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:18.811165 (+ 35us) server_negotiation.cc:408] Connection header received
0812 19:57:18.811235 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:18.811242 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:18.811485 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:18.811752 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:18.813021 (+ 1269us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53956: BlockingRecv error: recv got EOF from 127.0.0.1:53956 (error 108)
Metrics: {"server-negotiator.queue_time_us":24762,"thread_start_us":24513,"threads_started":1}
W20260812 19:57:18.816531 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19706: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:37683: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:18.816730 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19706
I20260812 19:57:18.842445 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37683
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=40855
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:19.235874 19872 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:19.236249 19872 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:19.236431 19872 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:57:19.236527 19872 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:19.245261 19872 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:19.245582 19872 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:19.257779 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40855
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:19.259622 19872 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:19.261165 19872 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:19.272914 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:19.274041 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:19.274075 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:19.274619 19872 server_base.cc:1061] running on GCE node
I20260812 19:57:19.275256 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:19.275408 19872 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:19.275496 19872 hybrid_clock.cc:648] HybridClock initialized: now 1786564639275461 us; error 0 us; skew 500 ppm
I20260812 19:57:19.275775 19872 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:19 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564639, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:19.279804 19872 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:19.281749 19872 webserver.cc:533] Webserver started at http://127.25.14.1:40855/ using document root <none> and password file <none>
I20260812 19:57:19.282742 19872 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:19.282881 19872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:19.289780 19872 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:57:19.293279 19887 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:19.295141 19872 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:19.295430 19872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:19.296306 19872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:19.330386 19872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:19.334808 19872 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:19.335141 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:19.336278 19872 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:19.338230 19872 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:19.338356 19872 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:19.338518 19872 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:19.338595 19872 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:57:19.339046 19872 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:57:19.339139 19872 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:57:19.339233 19872 fault_injection.cc:59] injecting fault for kudu (pid 19872): FLAGS_fault_before_start (process will exit)
I20260812 19:57:19.370051 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37683
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=40855
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:19.761444 19897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:19.761847 19897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:19.762076 19897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:19.770972 19897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:19.771265 19897 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:19.783083 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=40855
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:19.785007 19897 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:19.786706 19897 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:19.800225 19906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:19.800028 19905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:19.801927 19908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:19.803009 19897 server_base.cc:1061] running on GCE node
I20260812 19:57:19.803717 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:19.803915 19897 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:19.804035 19897 hybrid_clock.cc:648] HybridClock initialized: now 1786564639803994 us; error 0 us; skew 500 ppm
I20260812 19:57:19.804369 19897 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:19 dist-test-slave-p3t5 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564639, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:19.808516 19897 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:19.810600 19897 webserver.cc:533] Webserver started at http://127.25.14.1:40855/ using document root <none> and password file <none>
I20260812 19:57:19.811455 19897 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:19.811604 19897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:19.817963 19897 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:19.821437 19915 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:19.823340 19897 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:19.823594 19897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:19.824452 19897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:19.868750 19897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:19.873466 19897 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:19.873790 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:19.874907 19897 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:19.876679 19897 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:19.876811 19897 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:19.876942 19897 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:19.877031 19897 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:19.916438 19897 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37683
I20260812 19:57:19.916493 20062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37683 every 8 connection(s)
I20260812 19:57:19.918267 19897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:19.918599 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 19897
I20260812 19:57:19.919006 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19897
Aug 12 19:57:19 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564639, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:19 dist-test-slave-p3t5 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564639, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
W20260812 19:57:19.947465 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19897: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:37683: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:19.947670 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19897
I20260812 19:57:19.972082 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19285
W20260812 19:57:19.973313 19899 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:19.922780 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48209 (local address 127.25.14.62:43693)
0812 19:57:19.923037 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:19.923046 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:19.936252 (+ 13206us) server_negotiation.cc:408] Connection header received
0812 19:57:19.936611 (+ 359us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:19.936622 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:19.936733 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:19.936819 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:19.942458 (+ 5639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.943999 (+ 1541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:57:19.945893 (+ 1894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.946455 (+ 562us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:57:19.972398 (+ 25943us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48209: BlockingRecv error: recv got EOF from 127.25.14.1:48209 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260812 19:57:19.973543 20072 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:19.924855 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:55483 (local address 127.25.14.61:40793)
0812 19:57:19.930345 (+ 5490us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:19.930349 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:19.930368 (+ 19us) server_negotiation.cc:408] Connection header received
0812 19:57:19.930541 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:19.930552 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:19.930674 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:19.930846 (+ 172us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:19.935670 (+ 4824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.937192 (+ 1522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:57:19.939717 (+ 2525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.940237 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:57:19.945205 (+ 4968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:57:19.945264 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:57:19.945271 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 19:57:19.945318 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:57:19.949250 (+ 3932us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 19:57:19.972140 (+ 22890us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:55483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":5382,"thread_start_us":5269,"threads_started":1}
W20260812 19:57:19.974643 20070 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:19.924328 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58205 (local address 127.25.14.60:45943)
0812 19:57:19.929376 (+ 5048us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:19.929385 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:19.929417 (+ 32us) server_negotiation.cc:408] Connection header received
0812 19:57:19.933817 (+ 4400us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:19.933824 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:19.933924 (+ 100us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:19.934068 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:19.935829 (+ 1761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.937426 (+ 1597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:57:19.940800 (+ 3374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:57:19.941373 (+ 573us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:57:19.945859 (+ 4486us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:57:19.945937 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:57:19.945945 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 19:57:19.945995 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:57:19.953497 (+ 7502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 19:57:19.973597 (+ 20100us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":4886,"thread_start_us":4665,"threads_started":1}
W20260812 19:57:19.976013 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19285: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:44205: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:19.976167 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19285
I20260812 19:57:20.001348 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19441
W20260812 19:57:20.004891 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19441: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:38079: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:20.005126 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19441
I20260812 19:57:20.029251 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19594
W20260812 19:57:20.032083 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 19594: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:43693: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:57:20.032203 19899 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:20.030595 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.25.14.62:43693)
0812 19:57:20.030805 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:20.030814 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:20.030837 (+ 23us) server_negotiation.cc:408] Connection header received
0812 19:57:20.030921 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:20.030926 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:20.031009 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:20.031132 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:20.031948 (+ 816us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52046: BlockingRecv error: recv got EOF from 127.0.0.1:52046 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 19:57:20.032253 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 19594
I20260812 19:57:20.061369 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18891
I20260812 19:57:20.062912 18958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:20.180045 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 18891
I20260812 19:57:20.209909 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 18977
I20260812 19:57:20.211288 19045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:20.328711 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 18977
Shutting down hive metastore.
Aug 12 19:57:20 dist-test-slave-p3t5 krb5kdc[20087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:57:20 dist-test-slave-p3t5 krb5kdc[20087](info): set up 2 sockets
Aug 12 19:57:20 dist-test-slave-p3t5 krb5kdc[20087](info): commencing operation
krb5kdc: starting...
W20260812 19:57:22.864704 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.123s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:57:23 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:57:24: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:57:26 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564646, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:57:31 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564651, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:57:31 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564651, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:33.051393 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:33.427245 20315 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:33.427605 20315 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:33.427691 20315 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:33.436502 20315 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:33.436668 20315 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:33.436769 20315 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:33.436826 20315 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:33.448487 20315 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:33.450366 20315 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:33.451903 20315 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:33.461171 20320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:33.461246 20321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:33.461424 20323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:33.467738 20315 server_base.cc:1061] running on GCE node
I20260812 19:57:33.468740 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:33.468972 20315 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:33.469084 20315 hybrid_clock.cc:648] HybridClock initialized: now 1786564653469059 us; error 0 us; skew 500 ppm
I20260812 19:57:33.469509 20315 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:33 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564653, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:33.473666 20315 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:57:33.475916 20315 webserver.cc:533] Webserver started at http://127.25.14.62:40971/ using document root <none> and password file <none>
I20260812 19:57:33.476775 20315 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:33.476928 20315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:33.483083 20315 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:57:33.486482 20329 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:33.488227 20315 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:33.488467 20315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:33.489313 20315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:33.509141 20315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:33.513762 20315 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:33.514078 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:33.537261 20315 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43693
I20260812 19:57:33.537281 20387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43693 every 8 connection(s)
I20260812 19:57:33.539143 20315 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:57:33.547960 20388 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:57:33.548964 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20315
I20260812 19:57:33.549199 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:57:33.554087 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap starting.
I20260812 19:57:33.576633 20388 log.cc:826] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:57:33.586998 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:57:33.587641 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap complete.
I20260812 19:57:33.595649 20388 raft_consensus.cc:359] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:33.596544 20388 raft_consensus.cc:740] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede2f0430904328b231f15c3425a9cf, State: Initialized, Role: FOLLOWER
I20260812 19:57:33.597527 20388 consensus_queue.cc:260] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:33.598628 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:33.598995 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:33.599988 20388 sys_catalog.cc:565] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:33.616915 20407 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:33.617055 20407 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: loading cluster ID for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:33.622495 20407 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: acquiring CA information for follower catalog manager: success
Aug 12 19:57:33 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564653, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:33.625892 20407 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:57:33.640067 20315 master_runner.cc:187] Error getting master registration for 127.25.14.61:40793: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40793: connect: Connection refused (error 111),
I20260812 19:57:33.647208 20315 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:33.695540 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40793
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:34.082731 20413 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:34.083093 20413 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:34.083290 20413 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:34.092162 20413 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:34.092289 20413 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:34.092386 20413 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:34.092480 20413 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:34.104048 20413 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:34.105887 20413 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:34.107403 20413 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:34.117367 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:34.117343 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:34.117837 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:34.118728 20413 server_base.cc:1061] running on GCE node
I20260812 19:57:34.119707 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:34.119877 20413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:34.119953 20413 hybrid_clock.cc:648] HybridClock initialized: now 1786564654119933 us; error 0 us; skew 500 ppm
I20260812 19:57:34.120291 20413 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564654, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:34.124626 20413 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:57:34.127091 20413 webserver.cc:533] Webserver started at http://127.25.14.61:42993/ using document root <none> and password file <none>
I20260812 19:57:34.127955 20413 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:34.128131 20413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:34.134742 20413 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:34.138042 20428 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:34.139657 20413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:57:34.139909 20413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "63916aa6f5da40a4bdf497abe2f04657"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:34.140753 20413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:34.191319 20413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:34.196081 20413 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:34.196412 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:34.219830 20413 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:40793
I20260812 19:57:34.219866 20489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:40793 every 8 connection(s)
I20260812 19:57:34.221799 20413 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:57:34.229483 20490 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:57:34.232048 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20413
I20260812 19:57:34.232307 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:57:34.233511 20490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Bootstrap starting.
I20260812 19:57:34.253538 20490 log.cc:826] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:34.266590 20490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:57:34.267354 20490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: Bootstrap complete.
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:57:34.275461 20490 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:34.276283 20490 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63916aa6f5da40a4bdf497abe2f04657, State: Initialized, Role: FOLLOWER
I20260812 19:57:34.277172 20490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:34.278155 20494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:34.278477 20494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:34.279458 20490 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:34.291229 20507 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:34.291478 20507 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: loading cluster ID for follower catalog manager: success
I20260812 19:57:34.297065 20507 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: acquiring CA information for follower catalog manager: success
I20260812 19:57:34.303368 20507 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:34.331496 20413 master_runner.cc:187] Error getting master registration for 127.25.14.60:45943: Network error: Client connection negotiation failed: client connection to 127.25.14.60:45943: connect: Connection refused (error 111),
I20260812 19:57:34.335749 20413 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:34.389881 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45943
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:57:34.776168 20514 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:34.776629 20514 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:34.776791 20514 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:34.786223 20514 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:34.786391 20514 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:34.786520 20514 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:34.786604 20514 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:34.798362 20514 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:34.800207 20514 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:34.801959 20514 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:34.811139 20519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:34.811944 20520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:34.812273 20523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:34.812775 20514 server_base.cc:1061] running on GCE node
I20260812 19:57:34.813668 20514 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:34.813885 20514 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:34.813994 20514 hybrid_clock.cc:648] HybridClock initialized: now 1786564654813969 us; error 0 us; skew 500 ppm
I20260812 19:57:34.814378 20514 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564654, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:34.818519 20514 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:57:34.820788 20514 webserver.cc:533] Webserver started at http://127.25.14.60:42137/ using document root <none> and password file <none>
I20260812 19:57:34.821717 20514 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:34.821887 20514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:34.827875 20514 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 19:57:34.831135 20529 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:34.832595 20514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:57:34.832798 20514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
format_stamp: "Formatted at 2026-08-12 19:57:14 on dist-test-slave-p3t5"
I20260812 19:57:34.833686 20514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:34.864776 20514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:34.869194 20514 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:34.869517 20514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:34.891724 20514 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:45943
I20260812 19:57:34.891745 20598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:45943 every 8 connection(s)
I20260812 19:57:34.893499 20514 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:57:34.896013 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20514
I20260812 19:57:34.896238 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:57:34.902341 20599 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:57:34.906352 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Bootstrap starting.
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:34.926473 20599 log.cc:826] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:34.937803 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:57:34.938398 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Bootstrap complete.
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:34.946465 20599 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:34.947335 20599 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d441cbaf4244a3d98862fb6f936f0bf, State: Initialized, Role: FOLLOWER
I20260812 19:57:34.948148 20599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:34.949134 20618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:34.949470 20618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:34.950417 20599 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:34.963807 20630 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:34.963927 20630 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: loading cluster ID for follower catalog manager: success
Aug 12 19:57:34 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564654, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:34.969591 20630 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: acquiring CA information for follower catalog manager: success
I20260812 19:57:34.972404 20630 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:57:35.002394 20514 master_runner.cc:264] No leader master found from master 3d441cbaf4244a3d98862fb6f936f0bf
I20260812 19:57:35.002579 20514 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:57:35.280544 20633 raft_consensus.cc:493] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:57:35.280989 20633 raft_consensus.cc:515] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:35.283429 20633 leader_election.cc:290] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63916aa6f5da40a4bdf497abe2f04657 (127.25.14.61:40793), 3d441cbaf4244a3d98862fb6f936f0bf (127.25.14.60:45943)
I20260812 19:57:35.293200 20460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fede2f0430904328b231f15c3425a9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63916aa6f5da40a4bdf497abe2f04657" is_pre_election: true
I20260812 19:57:35.293241 20566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fede2f0430904328b231f15c3425a9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" is_pre_election: true
I20260812 19:57:35.293908 20460 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fede2f0430904328b231f15c3425a9cf in term 1.
I20260812 19:57:35.293970 20566 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fede2f0430904328b231f15c3425a9cf in term 1.
I20260812 19:57:35.294706 20330 leader_election.cc:304] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63916aa6f5da40a4bdf497abe2f04657, fede2f0430904328b231f15c3425a9cf; no voters:
I20260812 19:57:35.295207 20633 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:57:35.295382 20633 raft_consensus.cc:493] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:57:35.295516 20633 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:57:35.299568 20633 raft_consensus.cc:515] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:35.300424 20633 leader_election.cc:290] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [CANDIDATE]: Term 2 election: Requested vote from peers 63916aa6f5da40a4bdf497abe2f04657 (127.25.14.61:40793), 3d441cbaf4244a3d98862fb6f936f0bf (127.25.14.60:45943)
I20260812 19:57:35.301121 20460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fede2f0430904328b231f15c3425a9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63916aa6f5da40a4bdf497abe2f04657"
I20260812 19:57:35.301162 20566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fede2f0430904328b231f15c3425a9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d441cbaf4244a3d98862fb6f936f0bf"
I20260812 19:57:35.301370 20460 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:57:35.301445 20566 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:57:35.305948 20460 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede2f0430904328b231f15c3425a9cf in term 2.
I20260812 19:57:35.306667 20330 leader_election.cc:304] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63916aa6f5da40a4bdf497abe2f04657, fede2f0430904328b231f15c3425a9cf; no voters:
I20260812 19:57:35.307178 20633 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:57:35.307821 20633 raft_consensus.cc:697] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 LEADER]: Becoming Leader. State: Replica: fede2f0430904328b231f15c3425a9cf, State: Running, Role: LEADER
I20260812 19:57:35.307847 20566 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede2f0430904328b231f15c3425a9cf in term 2.
I20260812 19:57:35.308561 20633 consensus_queue.cc:237] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:35.311928 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fede2f0430904328b231f15c3425a9cf. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.312271 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:35.313056 20639 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:57:35.315186 20639 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:57:35.316030 20639 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:35.316138 20639 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:57:35.317298 20639 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:57:35.318185 20639 catalog_manager.cc:6103] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Loaded TSK: 0
I20260812 19:57:35.320133 20639 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:35.498274 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:35.740196 20566 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Refusing update from remote peer fede2f0430904328b231f15c3425a9cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:57:35.741408 20637 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:57:35.745939 20645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fede2f0430904328b231f15c3425a9cf. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.746255 20645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:35.747509 20633 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.747769 20633 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:35.748759 20460 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 2 FOLLOWER]: Refusing update from remote peer fede2f0430904328b231f15c3425a9cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:57:35.748939 20645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.749181 20645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:35.749769 20637 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:57:35.753728 20653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.754181 20653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:35.753989 20649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fede2f0430904328b231f15c3425a9cf. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:35.754364 20649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:57:35.882298 20644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:35.882652 20644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:35.882885 20644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:35.891775 20644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:35.892038 20644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:35.903416 20644 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:35.905294 20644 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:35.906857 20644 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:35.917977 20658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:35.917937 20659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:35.918102 20662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:35.919481 20644 server_base.cc:1061] running on GCE node
I20260812 19:57:35.920234 20644 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:35.920423 20644 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:35.920548 20644 hybrid_clock.cc:648] HybridClock initialized: now 1786564655920505 us; error 0 us; skew 500 ppm
I20260812 19:57:35.920909 20644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:35 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:35.924901 20644 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:35.926879 20644 webserver.cc:533] Webserver started at http://127.25.14.1:37557/ using document root <none> and password file <none>
I20260812 19:57:35.927672 20644 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:35.927770 20644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:35.933917 20644 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:35.937207 20669 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:35.939013 20644 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:35.939229 20644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:35.940102 20644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:35.962231 20644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:35.966650 20644 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:35.967000 20644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:35.968098 20644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:35.969905 20644 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:35.970052 20644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:35.970182 20644 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:35.970273 20644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:36.010013 20644 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41055
I20260812 19:57:36.010107 20805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41055 every 8 connection(s)
I20260812 19:57:36.011698 20644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:36.014014 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20644
I20260812 19:57:36.014312 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:57:36.051812 20809 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:36.052165 20806 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:36.052346 20806 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.052345 20809 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.053699 20806 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:36.053714 20809 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:36.054199 20808 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:36.054360 20808 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.054950 20808 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:36.056751 20449 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:36.056751 20347 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:36.056905 20555 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:36.059023 20347 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:47865
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:36.174106 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:36.231307 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fede2f0430904328b231f15c3425a9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:36.231622 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: LEADER
W20260812 19:57:36.565900 20821 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:36.566311 20821 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:36.566505 20821 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:36.575824 20821 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:36.576090 20821 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:57:36.587539 20821 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:36.590077 20821 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:36.591775 20821 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:36.603199 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:36.603226 20827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:36.603974 20833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:36.604292 20821 server_base.cc:1061] running on GCE node
I20260812 19:57:36.604996 20821 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:36.605185 20821 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:36.605299 20821 hybrid_clock.cc:648] HybridClock initialized: now 1786564656605257 us; error 0 us; skew 500 ppm
I20260812 19:57:36.605695 20821 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:36.610148 20821 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:57:36.612088 20821 webserver.cc:533] Webserver started at http://127.25.14.2:34067/ using document root <none> and password file <none>
I20260812 19:57:36.612913 20821 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:36.613067 20821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:36.619364 20821 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 19:57:36.622522 20841 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:36.624121 20821 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:57:36.624323 20821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "2db9537736fa4ba289b33efdfb400a74"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:36.625133 20821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:36.683897 20821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:36.688138 20821 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:36.688448 20821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:36.689563 20821 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:36.691358 20821 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:36.691483 20821 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:36.691624 20821 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:36.691721 20821 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:36.729210 20821 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:33513
I20260812 19:57:36.729266 20970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:33513 every 8 connection(s)
I20260812 19:57:36.731122 20821 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:57:36.741142 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20821
I20260812 19:57:36.741375 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:36 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:36.768347 20974 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:36.768867 20974 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.769556 20971 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:36.768395 20972 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:36.769773 20971 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.769861 20972 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:36.770139 20974 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:36.770467 20971 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:36.771477 20972 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:36.772215 20449 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:33513)
I20260812 19:57:36.772215 20555 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:33513)
I20260812 19:57:36.773360 20347 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:33513)
I20260812 19:57:36.774326 20347 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:35957
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:57:36.897763 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:37.062119 20808 heartbeater.cc:499] Master 127.25.14.62:43693 was elected leader, sending a full tablet report...
W20260812 19:57:37.277863 20988 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:37.278277 20988 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:37.278439 20988 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:37.287395 20988 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:37.287694 20988 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:57:37.299404 20988 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:37.301312 20988 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:37.302882 20988 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:37.314112 20996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:37.314088 20999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:37.314164 20997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:37.315158 20988 server_base.cc:1061] running on GCE node
I20260812 19:57:37.315874 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:37.316074 20988 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:37.316182 20988 hybrid_clock.cc:648] HybridClock initialized: now 1786564657316141 us; error 0 us; skew 500 ppm
I20260812 19:57:37.316520 20988 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:37 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564657, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:37.320482 20988 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:57:37.322510 20988 webserver.cc:533] Webserver started at http://127.25.14.3:43185/ using document root <none> and password file <none>
I20260812 19:57:37.323354 20988 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:37.323516 20988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:37.329903 20988 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:57:37.333076 21006 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:37.334676 20988 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:57:37.334889 20988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "e62b5dc6447f458f9135f070b3e771a4"
format_stamp: "Formatted at 2026-08-12 19:57:16 on dist-test-slave-p3t5"
I20260812 19:57:37.335654 20988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:37.354246 20988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:37.358608 20988 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:37.358940 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:37.360064 20988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:37.361887 20988 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:37.361979 20988 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:37.362149 20988 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:37.362239 20988 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:37.399781 20988 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:45181
I20260812 19:57:37.399887 21132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:45181 every 8 connection(s)
I20260812 19:57:37.401679 20988 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:57:37.403728 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 20988
I20260812 19:57:37.403991 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:57:37 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564657, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:57:37 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564657, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:37 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564657, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:57:37.449540 21135 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:37.450062 21135 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:37.451081 21135 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:37.451627 21133 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:37.451822 21133 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:37.452512 21133 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:37.453266 20449 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:45181)
I20260812 19:57:37.453532 21134 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:37.453738 21134 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:37.454077 20555 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:45181)
I20260812 19:57:37.454337 21134 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:37.455942 20346 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:45181)
I20260812 19:57:37.457558 20346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:55337
I20260812 19:57:37.459439 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:57:37.468425 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:57:37.468654 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20315
I20260812 19:57:37.469733 20382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:37.584935 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20315
I20260812 19:57:37.615931 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--webserver_interface=127.25.14.62
--webserver_port=40971
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:37.777386 20972 heartbeater.cc:499] Master 127.25.14.62:43693 was elected leader, sending a full tablet report...
W20260812 19:57:37.779362 20972 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:43693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111)
W20260812 19:57:38.003933 21145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:38.004372 21145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:38.004540 21145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:38.013371 21145 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:57:38.013546 21145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:38.013701 21145 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:57:38.013792 21145 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:57:38.025159 21145 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=40971
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:38.027057 21145 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:38.028599 21145 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:38.038379 21155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:38.038726 21154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:38.038764 21157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:38.040091 21145 server_base.cc:1061] running on GCE node
I20260812 19:57:38.041102 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:38.041381 21145 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:38.041504 21145 hybrid_clock.cc:648] HybridClock initialized: now 1786564658041478 us; error 0 us; skew 500 ppm
I20260812 19:57:38.041967 21145 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564658, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:38.046625 21145 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:57:38.048982 21145 webserver.cc:533] Webserver started at http://127.25.14.62:40971/ using document root <none> and password file <none>
I20260812 19:57:38.049888 21145 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:38.050037 21145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:38.056303 21145 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 19:57:38.059760 21164 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:38.061600 21145 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 19:57:38.061863 21145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "fede2f0430904328b231f15c3425a9cf"
format_stamp: "Formatted at 2026-08-12 19:57:13 on dist-test-slave-p3t5"
I20260812 19:57:38.062880 21145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 19:57:38.065845 20808 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:43693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111)
I20260812 19:57:38.079128 21145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:38.089311 21145 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:38.089728 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:38.112699 21145 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43693
I20260812 19:57:38.112720 21221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43693 every 8 connection(s)
I20260812 19:57:38.114619 21145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:57:38.122493 21222 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:57:38.122577 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 21145
I20260812 19:57:38.122967 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20644
I20260812 19:57:38.127449 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap starting.
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:57:38.145722 20801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:38.149889 21222 log.cc:826] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:57:38.162890 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:57:38.163710 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: Bootstrap complete.
I20260812 19:57:38.174621 21222 raft_consensus.cc:359] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.175683 21222 raft_consensus.cc:740] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede2f0430904328b231f15c3425a9cf, State: Initialized, Role: FOLLOWER
I20260812 19:57:38.176668 21222 consensus_queue.cc:260] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.177605 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:38.177989 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:38.179078 21222 sys_catalog.cc:565] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:57:38.201851 21239 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:38.202868 21239 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: loading cluster ID for follower catalog manager: success
I20260812 19:57:38.210924 21239 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: acquiring CA information for follower catalog manager: success
I20260812 19:57:38.216430 21239 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:57:38.255000 21145 master_runner.cc:264] No leader master found from master fede2f0430904328b231f15c3425a9cf
I20260812 19:57:38.255162 21145 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:57:38.283335 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20644
I20260812 19:57:38.309208 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41055
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=37557
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:38.460381 21134 heartbeater.cc:499] Master 127.25.14.62:43693 was elected leader, sending a full tablet report...
I20260812 19:57:38.472066 21184 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e62b5dc6447f458f9135f070b3e771a4" instance_seqno: 1786564657390621) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:48099; Asking this server to re-register.
I20260812 19:57:38.473249 21134 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:38.475302 21184 ts_manager.cc:194] Registered new tserver with Master: e62b5dc6447f458f9135f070b3e771a4 (127.25.14.3:45181)
W20260812 19:57:38.710254 21246 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:38.710697 21246 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:38.710886 21246 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:38.719528 21246 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:38.719817 21246 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:38.731246 21246 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37557
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:38.733057 21246 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:38.734683 21246 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:38.745899 21253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:38.745769 21256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:38.745767 21254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:38.746858 21246 server_base.cc:1061] running on GCE node
I20260812 19:57:38.747563 21246 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:38.747769 21246 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:38.747877 21246 hybrid_clock.cc:648] HybridClock initialized: now 1786564658747836 us; error 0 us; skew 500 ppm
I20260812 19:57:38.748260 21246 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564658, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:38.752677 21246 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:38.754863 21246 webserver.cc:533] Webserver started at http://127.25.14.1:37557/ using document root <none> and password file <none>
I20260812 19:57:38.755725 21246 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:38.755867 21246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:38.762236 21246 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:57:38.765389 21263 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:38.766960 21246 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:57:38.767168 21246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:38.768160 21246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:38.787261 21246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:38.787756 20972 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:38.788038 20972 heartbeater.cc:499] Master 127.25.14.62:43693 was elected leader, sending a full tablet report...
I20260812 19:57:38.789109 21184 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2db9537736fa4ba289b33efdfb400a74" instance_seqno: 1786564656720291) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:59733; Asking this server to re-register.
I20260812 19:57:38.789841 20972 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:38.791092 21184 ts_manager.cc:194] Registered new tserver with Master: 2db9537736fa4ba289b33efdfb400a74 (127.25.14.2:33513)
I20260812 19:57:38.792454 21246 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:38.792764 21246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:38.793972 21246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:38.795732 21246 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:38.795856 21246 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:38.796008 21246 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:38.796095 21246 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:38.799131 21275 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fede2f0430904328b231f15c3425a9cf)
I20260812 19:57:38.799371 21275 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.801285 21275 leader_election.cc:290] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693), 63916aa6f5da40a4bdf497abe2f04657 (127.25.14.61:40793)
I20260812 19:57:38.802561 20460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63916aa6f5da40a4bdf497abe2f04657" is_pre_election: true
I20260812 19:57:38.810631 21196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fede2f0430904328b231f15c3425a9cf" is_pre_election: true
I20260812 19:57:38.811234 21196 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d441cbaf4244a3d98862fb6f936f0bf in term 2.
I20260812 19:57:38.811913 20533 leader_election.cc:304] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d441cbaf4244a3d98862fb6f936f0bf, fede2f0430904328b231f15c3425a9cf; no voters: 63916aa6f5da40a4bdf497abe2f04657
I20260812 19:57:38.812498 21275 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:57:38.812677 21275 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader fede2f0430904328b231f15c3425a9cf)
I20260812 19:57:38.812810 21275 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:57:38.816715 21275 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.818140 21275 leader_election.cc:290] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [CANDIDATE]: Term 3 election: Requested vote from peers fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693), 63916aa6f5da40a4bdf497abe2f04657 (127.25.14.61:40793)
I20260812 19:57:38.818292 21196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fede2f0430904328b231f15c3425a9cf"
I20260812 19:57:38.818559 21196 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:57:38.819293 20460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63916aa6f5da40a4bdf497abe2f04657"
I20260812 19:57:38.823210 21196 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d441cbaf4244a3d98862fb6f936f0bf in term 3.
I20260812 19:57:38.823813 20533 leader_election.cc:304] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d441cbaf4244a3d98862fb6f936f0bf, fede2f0430904328b231f15c3425a9cf; no voters: 63916aa6f5da40a4bdf497abe2f04657
I20260812 19:57:38.824203 21275 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:57:38.824693 21275 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [term 3 LEADER]: Becoming Leader. State: Replica: 3d441cbaf4244a3d98862fb6f936f0bf, State: Running, Role: LEADER
I20260812 19:57:38.825181 21275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.827498 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d441cbaf4244a3d98862fb6f936f0bf. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:38.827879 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:38.828893 21345 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:57:38.832019 21345 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:57:38.833134 21345 catalog_manager.cc:1295] Loaded cluster ID: fb50211412c4425292083df3986c4006
I20260812 19:57:38.833247 21345 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:57:38.835016 21345 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:57:38.836207 21345 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf: Loaded TSK: 0
I20260812 19:57:38.837272 21345 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:57:38.845636 21246 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41055
I20260812 19:57:38.845692 21398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41055 every 8 connection(s)
I20260812 19:57:38.847447 21246 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:38.857282 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 21246
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564658, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564658, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:57:38 dist-test-slave-p3t5 krb5kdc[20087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564658, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:57:38.919590 21400 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43693
I20260812 19:57:38.920091 21400 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:38.921078 21399 heartbeater.cc:344] Connected to a master server at 127.25.14.60:45943
I20260812 19:57:38.921226 21399 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:38.921312 21400 heartbeater.cc:507] Master 127.25.14.62:43693 requested a full tablet report, sending...
I20260812 19:57:38.921887 21399 heartbeater.cc:507] Master 127.25.14.60:45943 requested a full tablet report, sending...
I20260812 19:57:38.923619 21401 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40793
I20260812 19:57:38.923776 21401 heartbeater.cc:461] Registering TS with master...
I20260812 19:57:38.924224 21401 heartbeater.cc:507] Master 127.25.14.61:40793 requested a full tablet report, sending...
I20260812 19:57:38.924340 21184 ts_manager.cc:194] Registered new tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:38.924459 20555 ts_manager.cc:194] Re-registered known tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:38.925679 20449 ts_manager.cc:194] Re-registered known tserver with Master: 4eebc99ef5a1480f8ae29b989cadabba (127.25.14.1:41055)
I20260812 19:57:38.926653 20555 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:44581
I20260812 19:57:38.945079 21410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fede2f0430904328b231f15c3425a9cf)
I20260812 19:57:38.945366 21410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } }
I20260812 19:57:38.947723 21410 leader_election.cc:290] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693), 3d441cbaf4244a3d98862fb6f936f0bf (127.25.14.60:45943)
I20260812 19:57:38.961100 21196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fede2f0430904328b231f15c3425a9cf" is_pre_election: true
I20260812 19:57:38.961660 21196 raft_consensus.cc:2393] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63916aa6f5da40a4bdf497abe2f04657 in current term 3: Already voted for candidate 3d441cbaf4244a3d98862fb6f936f0bf in this term.
I20260812 19:57:38.965960 20566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63916aa6f5da40a4bdf497abe2f04657" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" is_pre_election: true
I20260812 19:57:38.966924 20431 leader_election.cc:304] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63916aa6f5da40a4bdf497abe2f04657; no voters: 3d441cbaf4244a3d98862fb6f936f0bf, fede2f0430904328b231f15c3425a9cf
I20260812 19:57:38.967475 21410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:57:38.971153 21410 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:57:39.057443 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21246
W20260812 19:57:39.059122 21418 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:39.055034 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.25.14.1:41055)
0812 19:57:39.055754 (+ 720us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:39.055761 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:39.055797 (+ 36us) server_negotiation.cc:408] Connection header received
0812 19:57:39.055868 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:39.055875 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:39.056095 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:39.056301 (+ 206us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:39.057356 (+ 1055us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47254: BlockingRecv error: recv got EOF from 127.0.0.1:47254 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":362,"threads_started":1}
W20260812 19:57:39.060346 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21246: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:41055: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:39.060515 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 21246
W20260812 19:57:39.060535 21419 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:39.058485 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.25.14.1:41055)
0812 19:57:39.059097 (+ 612us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:39.059102 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:39.059124 (+ 22us) server_negotiation.cc:408] Connection header received
0812 19:57:39.059174 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:39.059179 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:39.059239 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:39.059374 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:39.060224 (+ 850us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47266: BlockingRecv error: recv got EOF from 127.0.0.1:47266 (error 108)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":381,"threads_started":1}
I20260812 19:57:39.084576 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41055
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=37557
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:39.226704 21196 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [term 3 FOLLOWER]: Refusing update from remote peer 3d441cbaf4244a3d98862fb6f936f0bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:57:39.227757 21330 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:57:39.232029 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d441cbaf4244a3d98862fb6f936f0bf. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.232452 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:39.233913 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.234234 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: LEADER
I20260812 19:57:39.235244 20460 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [term 3 FOLLOWER]: Refusing update from remote peer 3d441cbaf4244a3d98862fb6f936f0bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:57:39.235744 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.235945 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede2f0430904328b231f15c3425a9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:39.235992 21424 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:57:39.239308 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d441cbaf4244a3d98862fb6f936f0bf. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.239660 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:57:39.239313 21429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.239801 21429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63916aa6f5da40a4bdf497abe2f04657 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:57:39.478850 21420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:39.479230 21420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:39.479430 21420 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:57:39.479524 21420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:39.488480 21420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:39.488778 21420 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:39.499686 21420 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37557
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:39.501564 21420 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:39.503335 21420 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:39.514247 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:39.514261 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:39.514186 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:39.515488 21420 server_base.cc:1061] running on GCE node
I20260812 19:57:39.516240 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:39.516422 21420 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:39.516552 21420 hybrid_clock.cc:648] HybridClock initialized: now 1786564659516510 us; error 0 us; skew 500 ppm
I20260812 19:57:39.516906 21420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:39 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564659, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:39.520839 21420 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:39.522810 21420 webserver.cc:533] Webserver started at http://127.25.14.1:37557/ using document root <none> and password file <none>
I20260812 19:57:39.523693 21420 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:39.523824 21420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:39.529968 21420 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:57:39.533144 21446 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:39.534690 21420 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:57:39.534884 21420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:39.535746 21420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:39.554911 21420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:39.559587 21420 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:39.559962 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:39.561100 21420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:39.562987 21420 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:39.563103 21420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:39.563261 21420 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:39.563336 21420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:57:39.563740 21420 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:57:39.563828 21420 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:57:39.563943 21420 fault_injection.cc:59] injecting fault for kudu (pid 21420): FLAGS_fault_before_start (process will exit)
I20260812 19:57:39.591457 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:41055
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--webserver_port=37557
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:57:39.737679 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fede2f0430904328b231f15c3425a9cf" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43693 } } peers { permanent_uuid: "63916aa6f5da40a4bdf497abe2f04657" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40793 } } peers { permanent_uuid: "3d441cbaf4244a3d98862fb6f936f0bf" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 45943 } } }
I20260812 19:57:39.738034 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf [sys.catalog]: This master's current role is: LEADER
W20260812 19:57:39.973801 21455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:57:39.974210 21455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:57:39.974373 21455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:57:39.983253 21455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:57:39.983501 21455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:57:39.994488 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:41055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37557
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43693,127.25.14.61:40793,127.25.14.60:45943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:57:39.996330 21455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:57:39.997944 21455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:57:40.009608 21463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:40.010035 21466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:57:40.010476 21464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:57:40.011579 21455 server_base.cc:1061] running on GCE node
I20260812 19:57:40.012336 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:57:40.012504 21455 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:57:40.012579 21455 hybrid_clock.cc:648] HybridClock initialized: now 1786564660012559 us; error 0 us; skew 500 ppm
I20260812 19:57:40.012878 21455 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:57:40 dist-test-slave-p3t5 krb5kdc[20087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564660, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:57:40.017311 21455 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:57:40.019295 21455 webserver.cc:533] Webserver started at http://127.25.14.1:37557/ using document root <none> and password file <none>
I20260812 19:57:40.020162 21455 fs_manager.cc:362] Metadata directory not provided
I20260812 19:57:40.020284 21455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:57:40.026752 21455 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 19:57:40.029865 21472 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:40.031490 21455 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:57:40.031641 21455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4eebc99ef5a1480f8ae29b989cadabba"
format_stamp: "Formatted at 2026-08-12 19:57:15 on dist-test-slave-p3t5"
I20260812 19:57:40.032382 21455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:57:40.072753 21455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:57:40.076761 21455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:57:40.077010 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:57:40.078030 21455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:57:40.079675 21455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:57:40.079767 21455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:40.079849 21455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:57:40.079969 21455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:57:40.119454 21455 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41055
I20260812 19:57:40.119506 21597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41055 every 8 connection(s)
I20260812 19:57:40.121197 21455 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:57:40.129591 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 21455
I20260812 19:57:40.130016 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21455
W20260812 19:57:40.135349 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21455: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:41055: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:40.135522 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 21455
I20260812 19:57:40.160228 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20821
W20260812 19:57:40.161054 21612 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:40.126403 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:35439 (local address 127.25.14.60:45943)
0812 19:57:40.126981 (+ 578us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:40.126990 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:40.132148 (+ 5158us) server_negotiation.cc:408] Connection header received
0812 19:57:40.134201 (+ 2053us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:40.134210 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:40.134338 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:40.134447 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:40.160033 (+ 25586us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:35439: BlockingRecv error: recv got EOF from 127.25.14.1:35439 (error 108)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":248,"threads_started":1}
W20260812 19:57:40.161051 21608 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:40.125701 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:38275 (local address 127.25.14.62:43693)
0812 19:57:40.126268 (+ 567us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:40.126274 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:40.132876 (+ 6602us) server_negotiation.cc:408] Connection header received
0812 19:57:40.133782 (+ 906us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:40.133789 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:40.133874 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:40.134017 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:40.160077 (+ 26060us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:38275: BlockingRecv error: recv got EOF from 127.25.14.1:38275 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":940,"threads_started":1}
W20260812 19:57:40.161084 21607 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:40.125186 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:33473 (local address 127.25.14.61:40793)
0812 19:57:40.125789 (+ 603us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:40.125798 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:40.128080 (+ 2282us) server_negotiation.cc:408] Connection header received
0812 19:57:40.133766 (+ 5686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:40.133772 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:40.133847 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:40.133979 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:40.159975 (+ 25996us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:33473: BlockingRecv error: recv got EOF from 127.25.14.1:33473 (error 108)
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":250,"threads_started":1}
W20260812 19:57:40.163777 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20821: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:33513: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:40.164022 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20821
I20260812 19:57:40.190053 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20988
W20260812 19:57:40.193048 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20988: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:45181: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:57:40.193195 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20988
I20260812 19:57:40.216346 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21145
W20260812 19:57:40.219061 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21145: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:43693: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:57:40.219183 21608 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:57:40.217529 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.25.14.62:43693)
0812 19:57:40.217735 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0812 19:57:40.217743 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:57:40.217771 (+ 28us) server_negotiation.cc:408] Connection header received
0812 19:57:40.217985 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:57:40.217990 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:57:40.218056 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:57:40.218169 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:57:40.218929 (+ 760us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57216: BlockingRecv error: recv got EOF from 127.0.0.1:57216 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 19:57:40.219245 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 21145
I20260812 19:57:40.249212 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20413
I20260812 19:57:40.250582 20485 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 19:57:40.269069 20533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111)
W20260812 19:57:40.270864 20533 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3d441cbaf4244a3d98862fb6f936f0bf -> Peer fede2f0430904328b231f15c3425a9cf (127.25.14.62:43693): Couldn't send request to peer fede2f0430904328b231f15c3425a9cf. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:43693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:57:40.365386 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20413
I20260812 19:57:40.394156 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 20514
I20260812 19:57:40.395499 20594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:57:40.510677 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 20514
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/9 (47542 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/10
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:57:41 dist-test-slave-p3t5 krb5kdc[21632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:57:41 dist-test-slave-p3t5 krb5kdc[21632](info): set up 2 sockets
Aug 12 19:57:41 dist-test-slave-p3t5 krb5kdc[21632](info): commencing operation
krb5kdc: starting...
W20260812 19:57:43.214126 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.261s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:57:43 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:57:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:57:43Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:57:51: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:57:54 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564674, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:00 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564680, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:00 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564680, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:00.593815 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:00.983717 21975 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:00.984180 21975 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:00.984338 21975 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:00.993018 21975 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:00.993208 21975 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:00.993338 21975 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:00.993433 21975 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:01.005043 21975 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:01.007014 21975 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:01.008646 21975 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:01.018551 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:01.018610 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:01.019336 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:01.019464 21975 server_base.cc:1061] running on GCE node
I20260812 19:58:01.020445 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:01.022115 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:01.023414 21975 hybrid_clock.cc:648] HybridClock initialized: now 1786564681023365 us; error 44 us; skew 500 ppm
I20260812 19:58:01.023872 21975 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:01.027549 21975 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:01.029361 21975 webserver.cc:533] Webserver started at http://127.25.14.62:35599/ using document root <none> and password file <none>
I20260812 19:58:01.030222 21975 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:01.030326 21975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:01.030678 21975 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:01.033226 21975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.034152 21975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.039965 21975 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:58:01.043741 21992 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:01.045219 21975 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:58:01.045430 21975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.045704 21975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:01.074227 21975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:01.085115 21975 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:01.085471 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:01.108510 21975 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35465
I20260812 19:58:01.108534 22051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35465 every 8 connection(s)
I20260812 19:58:01.110340 21975 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:01.110744 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 21975
I20260812 19:58:01.110993 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:58:01.115895 22052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:01.122546 22052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
I20260812 19:58:01.149717 22052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
W20260812 19:58:01.151340 21997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
W20260812 19:58:01.153138 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:01.202553 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } attempt: 1
W20260812 19:58:01.205080 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:01.265043 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46865
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:01.273365 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } attempt: 2
W20260812 19:58:01.275789 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
I20260812 19:58:01.367132 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } attempt: 3
W20260812 19:58:01.369419 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
I20260812 19:58:01.512737 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } attempt: 4
W20260812 19:58:01.515481 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111)
W20260812 19:58:01.648144 22062 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:01.648597 22062 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:01.648768 22062 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:01.658015 22062 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:01.658186 22062 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:01.658306 22062 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:01.658401 22062 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:01.670302 22062 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:01.672154 22062 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:01.673799 22062 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:01.686457 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:01.686501 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:01.687892 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:01.688310 22062 server_base.cc:1061] running on GCE node
I20260812 19:58:01.689294 22062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:01.691088 22062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:01.692559 22062 hybrid_clock.cc:648] HybridClock initialized: now 1786564681692473 us; error 90 us; skew 500 ppm
I20260812 19:58:01.692976 22062 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:01.696461 22062 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:58:01.698364 22062 webserver.cc:533] Webserver started at http://127.25.14.61:37699/ using document root <none> and password file <none>
I20260812 19:58:01.699138 22062 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:01.699239 22062 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:01.699573 22062 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:01.702349 22062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "139f449766ad4191bb0595834283db3b"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.703212 22062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "139f449766ad4191bb0595834283db3b"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.709367 22062 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:58:01.713497 22078 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:01.715242 22062 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:58:01.715477 22062 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "139f449766ad4191bb0595834283db3b"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:01.715703 22062 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:01.736333 22062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:01.740639 22062 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:01.740988 22062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:01.764662 22062 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46865
I20260812 19:58:01.764705 22138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46865 every 8 connection(s)
I20260812 19:58:01.766577 22062 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:58:01.771952 22139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:01.772042 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22062
I20260812 19:58:01.772225 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:58:01.782229 22139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
I20260812 19:58:01.808071 22139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
I20260812 19:58:01.814730 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } attempt: 5
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:01 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:01.828723 22139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:58:01.830129 22084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
W20260812 19:58:01.832139 22139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:01.832660 22052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:58:01.834672 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:01.881498 22139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 1
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
W20260812 19:58:01.883891 22139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:01.885896 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
W20260812 19:58:01.888067 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:01.927363 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40063
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:01.952220 22139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 2
W20260812 19:58:01.954694 22139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:01.956322 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 2
W20260812 19:58:01.958386 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:02.046036 22139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 3
W20260812 19:58:02.048445 22139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:02.064713 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 3
W20260812 19:58:02.067121 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:02.191798 22139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 4
W20260812 19:58:02.194305 22139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
I20260812 19:58:02.244434 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 4
W20260812 19:58:02.246629 22052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111)
W20260812 19:58:02.324085 22153 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:02.324482 22153 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:02.324574 22153 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:02.333868 22153 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:02.333995 22153 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:02.334054 22153 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:02.334113 22153 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:02.345731 22153 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:02.347548 22153 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:02.349081 22153 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:02.358186 22158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:02.358214 22159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:02.358309 22162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:02.359558 22153 server_base.cc:1061] running on GCE node
I20260812 19:58:02.360492 22153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:02.362226 22153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:02.363512 22153 hybrid_clock.cc:648] HybridClock initialized: now 1786564682363472 us; error 54 us; skew 500 ppm
I20260812 19:58:02.363968 22153 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564682, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:02.367729 22153 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:58:02.369728 22153 webserver.cc:533] Webserver started at http://127.25.14.60:33399/ using document root <none> and password file <none>
I20260812 19:58:02.370549 22153 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:02.370658 22153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:02.371009 22153 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:02.373771 22153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "0a43729acacb43ac9a3089b405d536a1"
format_stamp: "Formatted at 2026-08-12 19:58:02 on dist-test-slave-p3t5"
I20260812 19:58:02.374647 22153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "0a43729acacb43ac9a3089b405d536a1"
format_stamp: "Formatted at 2026-08-12 19:58:02 on dist-test-slave-p3t5"
I20260812 19:58:02.380769 22153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:58:02.384956 22169 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:02.386700 22153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 19:58:02.386904 22153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "0a43729acacb43ac9a3089b405d536a1"
format_stamp: "Formatted at 2026-08-12 19:58:02 on dist-test-slave-p3t5"
I20260812 19:58:02.387108 22153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:02.426889 22153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:02.432030 22153 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:02.432329 22153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:02.455199 22153 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:40063
I20260812 19:58:02.455233 22233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:40063 every 8 connection(s)
I20260812 19:58:02.456964 22153 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:58:02.461985 22234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:02.466140 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22153
I20260812 19:58:02.466387 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:58:02.469813 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564682, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:02.493701 22139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 5
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:02.498550 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564682, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:02.521109 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Bootstrap starting.
I20260812 19:58:02.521822 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:58:02.523941 22052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } attempt: 5
I20260812 19:58:02.525954 22139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:02.527552 22139 log.cc:826] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Log is configured to *not* fsync() on all Append() calls
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564682, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:02 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564681, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:02.537309 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: No bootstrap required, opened a new log
I20260812 19:58:02.544642 22139 raft_consensus.cc:359] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.544952 22139 raft_consensus.cc:385] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:02.545105 22139 raft_consensus.cc:740] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139f449766ad4191bb0595834283db3b, State: Initialized, Role: FOLLOWER
I20260812 19:58:02.545981 22139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.546559 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Bootstrap starting.
I20260812 19:58:02.547093 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.547453 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.548545 22139 sys_catalog.cc:565] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:02.550694 22234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:02.551963 22234 log.cc:826] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:02.552516 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap starting.
I20260812 19:58:02.555649 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: No bootstrap required, opened a new log
I20260812 19:58:02.560551 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:02.562525 22052 log.cc:826] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:02.564043 22234 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.564499 22234 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:02.564666 22234 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a43729acacb43ac9a3089b405d536a1, State: Initialized, Role: FOLLOWER
I20260812 19:58:02.565577 22234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.576937 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: No bootstrap required, opened a new log
I20260812 19:58:02.584997 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.585992 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.586632 22234 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:02.589730 22052 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.590323 22052 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:02.593324 22052 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4307db228843427aa13bef0bf7bfca2d, State: Initialized, Role: FOLLOWER
I20260812 19:58:02.594519 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.598760 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.601038 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.600906 22052 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:02.597976 22062 master_runner.cc:187] Error getting master registration for 127.25.14.62:35465: OK, instance_id { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" instance_seqno: 1786564681102451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:58:02.605293 22062 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:58:02.610525 22268 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:58:02.610725 22268 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:58:02.614902 22283 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:58:02.615087 22283 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:58:02.611370 22153 master_runner.cc:187] Error getting master registration for 127.25.14.62:35465: OK, instance_id { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" instance_seqno: 1786564681102451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:58:02.618402 22153 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:02.634527 21975 master_runner.cc:264] No leader master found from master 4307db228843427aa13bef0bf7bfca2d
I20260812 19:58:02.634708 21975 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:58:02.636268 22294 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:58:02.636454 22294 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:58:02.685038 22176 tablet.cc:2406] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:58:02.724004 22258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:58:02.724386 22258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.726809 22258 leader_election.cc:290] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465), 139f449766ad4191bb0595834283db3b (127.25.14.61:46865)
I20260812 19:58:02.728469 22024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4307db228843427aa13bef0bf7bfca2d" is_pre_election: true
I20260812 19:58:02.729146 22024 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a43729acacb43ac9a3089b405d536a1 in term 0.
I20260812 19:58:02.730618 22111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139f449766ad4191bb0595834283db3b" is_pre_election: true
I20260812 19:58:02.731181 22111 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a43729acacb43ac9a3089b405d536a1 in term 0.
I20260812 19:58:02.732018 22170 leader_election.cc:304] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a43729acacb43ac9a3089b405d536a1, 4307db228843427aa13bef0bf7bfca2d; no voters:
I20260812 19:58:02.733319 22258 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:58:02.733592 22258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:58:02.733814 22258 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:02.737473 22258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
W20260812 19:58:02.744087 22086 tablet.cc:2406] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:58:02.745251 22024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4307db228843427aa13bef0bf7bfca2d"
I20260812 19:58:02.745517 22024 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:02.749761 22024 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a43729acacb43ac9a3089b405d536a1 in term 1.
I20260812 19:58:02.750552 22170 leader_election.cc:304] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a43729acacb43ac9a3089b405d536a1, 4307db228843427aa13bef0bf7bfca2d; no voters:
I20260812 19:58:02.753206 22111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139f449766ad4191bb0595834283db3b"
I20260812 19:58:02.753465 22111 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:02.757279 22111 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a43729acacb43ac9a3089b405d536a1 in term 1.
I20260812 19:58:02.758338 22258 leader_election.cc:290] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465), 139f449766ad4191bb0595834283db3b (127.25.14.61:46865)
I20260812 19:58:02.758841 22258 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:58:02.759375 22258 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a43729acacb43ac9a3089b405d536a1, State: Running, Role: LEADER
I20260812 19:58:02.760218 22258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:02.790293 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a43729acacb43ac9a3089b405d536a1. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.792506 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:02.800701 22299 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:58:02.803493 22299 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:58:02.820223 22111 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Refusing update from remote peer 0a43729acacb43ac9a3089b405d536a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:58:02.821540 22296 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:58:02.829559 22024 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Refusing update from remote peer 0a43729acacb43ac9a3089b405d536a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:58:02.830947 22296 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 19:58:02.839118 22000 tablet.cc:2406] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:58:02.856500 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a43729acacb43ac9a3089b405d536a1. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.856806 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.861011 22300 mvcc.cc:204] Tried to move back new op lower bound from 7317768940816973824 to 7317768940673409024. Current Snapshot: MvccSnapshot[applied={T|T < 7317768940816973824}]
I20260812 19:58:02.874430 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.874779 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.878659 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a43729acacb43ac9a3089b405d536a1. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.879004 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.881305 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.881763 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:02.883101 22299 catalog_manager.cc:1383] Generated new cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:02.883214 22299 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:58:02.890776 22303 mvcc.cc:204] Tried to move back new op lower bound from 7317768940816973824 to 7317768940673409024. Current Snapshot: MvccSnapshot[applied={T|T < 7317768940816973824}]
I20260812 19:58:02.904968 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.906402 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:02.917379 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:02.917649 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:02.931774 22299 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:58:02.933781 22299 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:58:02.958966 22299 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Generated new TSK 0
I20260812 19:58:02.960286 22299 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 19:58:03.102607 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.130s user 0.000s sys 0.030s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 19:58:03.285451 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.183s user 0.000s sys 0.032s
I20260812 19:58:03.286671 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:03.612927 22268 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:03.613036 22268 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: loading cluster ID for follower catalog manager: success
I20260812 19:58:03.615893 22268 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: acquiring CA information for follower catalog manager: success
I20260812 19:58:03.618167 22268 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:58:03.638701 22294 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:03.638823 22294 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: loading cluster ID for follower catalog manager: success
I20260812 19:58:03.641508 22294 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: acquiring CA information for follower catalog manager: success
I20260812 19:58:03.643530 22294 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:58:03.689369 22317 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:03.689931 22317 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:03.690119 22317 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:03.699107 22317 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:03.699374 22317 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:03.710923 22317 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:03.712719 22317 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:03.714295 22317 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:03.725013 22322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:03.725112 22323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:03.726042 22325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:03.726339 22317 server_base.cc:1061] running on GCE node
I20260812 19:58:03.727021 22317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:03.727960 22317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:03.729203 22317 hybrid_clock.cc:648] HybridClock initialized: now 1786564683729155 us; error 42 us; skew 500 ppm
I20260812 19:58:03.729545 22317 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:03 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:03.733134 22317 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:03.735091 22317 webserver.cc:533] Webserver started at http://127.25.14.1:36533/ using document root <none> and password file <none>
I20260812 19:58:03.735924 22317 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:03.736073 22317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:03.736477 22317 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:03.739270 22317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:03.740191 22317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:03.747128 22317 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:58:03.751192 22332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:03.752858 22317 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:03.753059 22317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:03.753273 22317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:03.769340 22317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:03.774466 22317 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:03.774817 22317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:03.776051 22317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:03.778043 22317 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:03.778156 22317 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:03.778316 22317 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:03.778393 22317 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:03.817399 22317 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38555
I20260812 19:58:03.817446 22463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38555 every 8 connection(s)
I20260812 19:58:03.819010 22317 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:03.820643 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22317
I20260812 19:58:03.820852 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:58:03 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:03 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:03 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:03.863693 22466 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:03.864197 22466 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:03.865453 22466 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:03.866410 22465 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:03.866566 22465 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:03.867152 22465 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:03.867525 22464 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:03.867653 22464 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:03.868139 22464 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:03.868520 22100 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
I20260812 19:58:03.869143 22013 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
I20260812 19:58:03.870806 22188 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
I20260812 19:58:03.873075 22188 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:41165
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:03.994668 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:04.380282 22484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:04.380654 22484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:04.380748 22484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:04.389529 22484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:04.389770 22484 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:58:04.402289 22484 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:04.404139 22484 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:04.405642 22484 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:04.416707 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:04.416682 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:04.417409 22495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:04.417939 22484 server_base.cc:1061] running on GCE node
I20260812 19:58:04.418574 22484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:04.419605 22484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:04.420843 22484 hybrid_clock.cc:648] HybridClock initialized: now 1786564684420782 us; error 40 us; skew 500 ppm
I20260812 19:58:04.421213 22484 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:04 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:04.424847 22484 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:58:04.426925 22484 webserver.cc:533] Webserver started at http://127.25.14.2:36211/ using document root <none> and password file <none>
I20260812 19:58:04.427906 22484 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:04.428022 22484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:04.428365 22484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:04.431232 22484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "1681a5ec2583418fb3fbc4c16989d31d"
format_stamp: "Formatted at 2026-08-12 19:58:04 on dist-test-slave-p3t5"
I20260812 19:58:04.432056 22484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "1681a5ec2583418fb3fbc4c16989d31d"
format_stamp: "Formatted at 2026-08-12 19:58:04 on dist-test-slave-p3t5"
I20260812 19:58:04.438728 22484 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 19:58:04.442667 22505 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:04.444339 22484 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:58:04.444593 22484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1681a5ec2583418fb3fbc4c16989d31d"
format_stamp: "Formatted at 2026-08-12 19:58:04 on dist-test-slave-p3t5"
I20260812 19:58:04.444841 22484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:04.486472 22484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:04.491008 22484 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:04.491351 22484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:04.492487 22484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:04.494413 22484 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:04.494526 22484 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:04.494689 22484 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:04.494766 22484 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:04.534375 22484 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:40941
I20260812 19:58:04.534456 22635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:40941 every 8 connection(s)
I20260812 19:58:04.536046 22484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:58:04.542502 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22484
I20260812 19:58:04.542758 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:58:04 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:04 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:04 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:04.585557 22638 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:04.585711 22637 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:04.586148 22637 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:04.587386 22637 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:04.586153 22638 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:04.589465 22638 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:04.589694 22013 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:40941)
I20260812 19:58:04.590942 22100 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:40941)
I20260812 19:58:04.591735 22636 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:04.591878 22636 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:04.592310 22636 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:04.593434 22188 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:40941)
I20260812 19:58:04.594314 22188 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:48079
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:04.718116 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:04.876143 22464 heartbeater.cc:499] Master 127.25.14.60:40063 was elected leader, sending a full tablet report...
W20260812 19:58:05.107831 22652 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:05.108214 22652 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:05.108418 22652 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:05.117375 22652 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:05.117703 22652 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:58:05.129529 22652 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:05.131423 22652 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:05.132915 22652 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:05.144650 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:05.144884 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:05.144682 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:05.146023 22652 server_base.cc:1061] running on GCE node
I20260812 19:58:05.146698 22652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:05.147774 22652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:05.149029 22652 hybrid_clock.cc:648] HybridClock initialized: now 1786564685148972 us; error 52 us; skew 500 ppm
I20260812 19:58:05.149389 22652 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:05.153277 22652 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:58:05.155313 22652 webserver.cc:533] Webserver started at http://127.25.14.3:46181/ using document root <none> and password file <none>
I20260812 19:58:05.156183 22652 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:05.156339 22652 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:05.156740 22652 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:05.159699 22652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "b97a2e411090422c99b6a839f1d73f72"
format_stamp: "Formatted at 2026-08-12 19:58:05 on dist-test-slave-p3t5"
I20260812 19:58:05.160607 22652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "b97a2e411090422c99b6a839f1d73f72"
format_stamp: "Formatted at 2026-08-12 19:58:05 on dist-test-slave-p3t5"
I20260812 19:58:05.166638 22652 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 19:58:05.170562 22668 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:05.172112 22652 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:58:05.172317 22652 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "b97a2e411090422c99b6a839f1d73f72"
format_stamp: "Formatted at 2026-08-12 19:58:05 on dist-test-slave-p3t5"
I20260812 19:58:05.172535 22652 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:05.187229 22652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:05.191684 22652 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:05.192022 22652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:05.193189 22652 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:05.195070 22652 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:05.195178 22652 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:05.195343 22652 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:05.195417 22652 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:05.236387 22652 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:45553
I20260812 19:58:05.236485 22799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:45553 every 8 connection(s)
I20260812 19:58:05.238067 22652 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:58:05.245138 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22652
I20260812 19:58:05.245378 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:05.272523 22802 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:05.272758 22803 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:05.273133 22802 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:05.273139 22803 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:05.274646 22802 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:05.275969 22803 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:05.276798 22013 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:45553)
I20260812 19:58:05.277261 22100 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:45553)
I20260812 19:58:05.282747 22801 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:05.282902 22801 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:05.283370 22801 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:05.284502 22188 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:45553)
I20260812 19:58:05.285651 22188 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:35387
I20260812 19:58:05.292071 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:05.301082 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:05.301283 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 21975
I20260812 19:58:05.302230 22045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:05.416786 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 21975
I20260812 19:58:05.446866 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--webserver_interface=127.25.14.62
--webserver_port=35599
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:05.593544 22637 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111)
I20260812 19:58:05.596913 22636 heartbeater.cc:499] Master 127.25.14.60:40063 was elected leader, sending a full tablet report...
W20260812 19:58:05.682859 22170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111)
W20260812 19:58:05.685257 22170 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 -> Peer 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465): Couldn't send request to peer 4307db228843427aa13bef0bf7bfca2d. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:58:05.842424 22815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:05.842876 22815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:05.843044 22815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:05.852200 22815 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:05.852370 22815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:05.852496 22815 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:05.852594 22815 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:05.864470 22815 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35599
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:05.866358 22815 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:05.867941 22815 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:05.877827 22465 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111)
W20260812 19:58:05.877849 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:05.878417 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:05.878808 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:05.880728 22815 server_base.cc:1061] running on GCE node
I20260812 19:58:05.881718 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:05.883265 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:05.884505 22815 hybrid_clock.cc:648] HybridClock initialized: now 1786564685884451 us; error 44 us; skew 500 ppm
I20260812 19:58:05.884896 22815 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:05.888504 22815 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:05.890473 22815 webserver.cc:533] Webserver started at http://127.25.14.62:35599/ using document root <none> and password file <none>
I20260812 19:58:05.891256 22815 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:05.891410 22815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:05.897415 22815 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:58:05.900600 22838 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:05.902350 22815 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:58:05.902570 22815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:05.903381 22815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:05.921427 22815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:05.925797 22815 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:05.926105 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:05.948815 22815 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35465
I20260812 19:58:05.948858 22896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35465 every 8 connection(s)
I20260812 19:58:05.950840 22815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:05.952877 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22815
I20260812 19:58:05.953276 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22317
I20260812 19:58:05.959869 22898 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:58:05.964957 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap starting.
Aug 12 19:58:05 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:58:05.979363 22458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:05.987825 22898 log.cc:826] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:05.998200 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:05.998827 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap complete.
I20260812 19:58:06.006307 22898 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:06.007181 22898 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4307db228843427aa13bef0bf7bfca2d, State: Initialized, Role: FOLLOWER
I20260812 19:58:06.008111 22898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:06.009271 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:06.009599 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:06.010564 22898 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:06.022306 22919 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:06.022527 22919 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: loading cluster ID for follower catalog manager: success
I20260812 19:58:06.027309 22919 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: acquiring CA information for follower catalog manager: success
I20260812 19:58:06.030224 22919 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564685, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:06.089306 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22317
I20260812 19:58:06.115157 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38555
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=36533
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:06.198350 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:06.198695 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:06.217991 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a43729acacb43ac9a3089b405d536a1. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:06.218331 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:06.288841 22801 heartbeater.cc:499] Master 127.25.14.60:40063 was elected leader, sending a full tablet report...
I20260812 19:58:06.294057 22855 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b97a2e411090422c99b6a839f1d73f72" instance_seqno: 1786564685225520) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:58421; Asking this server to re-register.
I20260812 19:58:06.295214 22802 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:06.297309 22855 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:45553)
W20260812 19:58:06.521293 22927 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:06.521684 22927 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:06.521867 22927 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:06.530465 22927 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:06.530718 22927 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:06.542544 22927 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36533
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:06.544371 22927 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:06.546060 22927 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:06.556993 22938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:06.556993 22940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:06.557680 22942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:06.558617 22927 server_base.cc:1061] running on GCE node
I20260812 19:58:06.559238 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:06.561563 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:06.562886 22927 hybrid_clock.cc:648] HybridClock initialized: now 1786564686562793 us; error 52 us; skew 500 ppm
I20260812 19:58:06.563375 22927 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564686, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:06.569159 22927 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:06.571985 22927 webserver.cc:533] Webserver started at http://127.25.14.1:36533/ using document root <none> and password file <none>
I20260812 19:58:06.572999 22927 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:06.573137 22927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:06.582921 22927 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 19:58:06.587718 22950 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:06.590031 22927 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 19:58:06.590261 22927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:06.591228 22927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:06.611886 22637 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:06.613236 22855 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1681a5ec2583418fb3fbc4c16989d31d" instance_seqno: 1786564684524814) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:43825; Asking this server to re-register.
I20260812 19:58:06.614071 22637 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:06.614787 22927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:06.615506 22855 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:40941)
I20260812 19:58:06.619760 22927 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:06.620062 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:06.621177 22927 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:06.623009 22927 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:06.623137 22927 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:06.623279 22927 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:06.623377 22927 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:06.664690 22927 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38555
I20260812 19:58:06.664755 23076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38555 every 8 connection(s)
I20260812 19:58:06.666487 22927 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:06.672453 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 22927
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564686, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:06.702232 23078 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:06.702718 23078 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:06.703889 23078 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:06.704778 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a43729acacb43ac9a3089b405d536a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:06.705063 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:06.706122 22855 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564686, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:06 dist-test-slave-p3t5 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564686, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:06.757066 23077 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:06.757274 23077 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:06.757961 23077 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:06.759744 22188 ts_manager.cc:194] Re-registered known tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
I20260812 19:58:06.761078 22188 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:52875
I20260812 19:58:06.761687 23079 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:06.761894 23079 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:06.762594 23079 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:06.766744 22100 ts_manager.cc:194] Re-registered known tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:38555)
I20260812 19:58:06.902587 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22927
W20260812 19:58:06.903978 23093 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:06.899678 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.25.14.1:38555)
0812 19:58:06.900588 (+ 910us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:06.900596 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:06.900627 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:58:06.900695 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:06.900703 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:06.900934 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:06.901182 (+ 248us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:06.902392 (+ 1210us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43780: BlockingRecv error: recv got EOF from 127.0.0.1:43780 (error 108)
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":505,"threads_started":1}
W20260812 19:58:06.905848 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22927: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:38555: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:58:06.905964 23093 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:06.903590 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.25.14.1:38555)
0812 19:58:06.904614 (+ 1024us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:06.904618 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:06.904634 (+ 16us) server_negotiation.cc:408] Connection header received
0812 19:58:06.904683 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:06.904687 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:06.904746 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:06.904829 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:06.905742 (+ 913us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43782: BlockingRecv error: recv got EOF from 127.0.0.1:43782 (error 108)
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":784,"threads_started":1}
I20260812 19:58:06.906073 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22927
I20260812 19:58:06.941509 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38555
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=36533
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:07.337432 23096 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:07.337924 23096 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:07.338106 23096 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:58:07.338208 23096 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:07.347240 23096 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:07.347533 23096 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:07.359138 23096 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36533
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:07.361003 23096 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:07.362679 23096 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:07.372992 23101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:07.373834 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:07.375274 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:07.375700 23096 server_base.cc:1061] running on GCE node
I20260812 19:58:07.376502 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:07.377573 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:07.378870 23096 hybrid_clock.cc:648] HybridClock initialized: now 1786564687378774 us; error 66 us; skew 500 ppm
I20260812 19:58:07.379215 23096 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:07 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564687, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:07.383088 23096 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:07.385205 23096 webserver.cc:533] Webserver started at http://127.25.14.1:36533/ using document root <none> and password file <none>
I20260812 19:58:07.386204 23096 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:07.386297 23096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:07.392638 23096 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 19:58:07.395901 23114 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:07.397451 23096 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:58:07.397671 23096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:07.398483 23096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:07.415648 23096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:07.420341 23096 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:07.420699 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:07.422117 23096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:07.424454 23096 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:07.424538 23096 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:07.424724 23096 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:07.424777 23096 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:58:07.425202 23096 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:58:07.425297 23096 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:58:07.425400 23096 fault_injection.cc:59] injecting fault for kudu (pid 23096): FLAGS_fault_before_start (process will exit)
I20260812 19:58:07.452414 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:38555
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=36533
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:07.840080 23124 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:07.840438 23124 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:07.840533 23124 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:07.849026 23124 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:07.849236 23124 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:07.860657 23124 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36533
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:07.862440 23124 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:07.863903 23124 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:07.875111 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:07.875067 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:07.875823 23137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:07.876287 23124 server_base.cc:1061] running on GCE node
I20260812 19:58:07.876888 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:07.877893 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:07.879110 23124 hybrid_clock.cc:648] HybridClock initialized: now 1786564687879046 us; error 53 us; skew 500 ppm
I20260812 19:58:07.879465 23124 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:07 dist-test-slave-p3t5 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564687, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:07.883203 23124 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:07.885169 23124 webserver.cc:533] Webserver started at http://127.25.14.1:36533/ using document root <none> and password file <none>
I20260812 19:58:07.886106 23124 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:07.886214 23124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:07.892242 23124 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:58:07.895388 23146 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:07.896899 23124 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 19:58:07.897100 23124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:07.897948 23124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:07.912801 23124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:07.917138 23124 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:07.917451 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:07.918707 23124 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:07.920468 23124 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:07.920598 23124 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:07.920737 23124 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:07.920833 23124 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:07.959991 23124 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38555
I20260812 19:58:07.960031 23271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38555 every 8 connection(s)
I20260812 19:58:07.962109 23124 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:07.970132 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 23124
I20260812 19:58:07.970598 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23124
W20260812 19:58:07.980649 23281 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:07.972605 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.25.14.1:38555)
0812 19:58:07.973350 (+ 745us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:07.973363 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:07.974240 (+ 877us) server_negotiation.cc:408] Connection header received
0812 19:58:07.976509 (+ 2269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:07.976539 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:07.976935 (+ 396us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:07.977175 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:07.979140 (+ 1965us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43792: BlockingRecv error: recv got EOF from 127.0.0.1:43792 (error 108)
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":467,"threads_started":1}
W20260812 19:58:07.984012 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23124: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:38555: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:07.984248 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 23124
I20260812 19:58:08.005592 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22484
W20260812 19:58:08.006405 23283 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:07.976466 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:53977 (local address 127.25.14.60:40063)
0812 19:58:07.988187 (+ 11721us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:07.988192 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:08.005332 (+ 17140us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:53977: BlockingRecv error: recv got EOF from 127.25.14.1:53977 (error 108)
Metrics: {"server-negotiator.queue_time_us":11599,"thread_start_us":11499,"threads_started":1}
W20260812 19:58:08.006418 23276 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:07.966009 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:41045 (local address 127.25.14.61:46865)
0812 19:58:07.972340 (+ 6331us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:07.972348 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:07.975969 (+ 3621us) server_negotiation.cc:408] Connection header received
0812 19:58:07.977531 (+ 1562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:07.977538 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:07.977651 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:07.977769 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:07.981857 (+ 4088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:07.983600 (+ 1743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:08.005400 (+ 21800us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:41045: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":6130,"thread_start_us":7375,"threads_started":1}
W20260812 19:58:08.006408 23126 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:07.966171 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:35361 (local address 127.25.14.62:35465)
0812 19:58:07.966445 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:07.966455 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:07.975132 (+ 8677us) server_negotiation.cc:408] Connection header received
0812 19:58:07.977120 (+ 1988us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:07.977126 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:07.977215 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:07.977332 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:07.983156 (+ 5824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:07.984643 (+ 1487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:08.005412 (+ 20769us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:35361: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":103}
W20260812 19:58:08.009236 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22484: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:40941: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:08.009418 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22484
I20260812 19:58:08.034761 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22652
W20260812 19:58:08.037837 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22652: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:45553: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:08.038041 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22652
I20260812 19:58:08.062314 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22815
W20260812 19:58:08.065021 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22815: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:35465: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:58:08.065083 23126 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:08.063464 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.25.14.62:35465)
0812 19:58:08.063696 (+ 232us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:08.063702 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:08.063726 (+ 24us) server_negotiation.cc:408] Connection header received
0812 19:58:08.063845 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:08.063850 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:08.063923 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:08.064049 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:08.064881 (+ 832us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42324: BlockingRecv error: recv got EOF from 127.0.0.1:42324 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 19:58:08.065169 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22815
W20260812 19:58:08.095439 22170 connection.cc:573] client connection to 127.25.14.62:35465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:58:08.095813 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22062
I20260812 19:58:08.097204 22133 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 19:58:08.097671 22170 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 -> Peer 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465): Couldn't send request to peer 4307db228843427aa13bef0bf7bfca2d. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:58:08.211310 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22062
I20260812 19:58:08.240814 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 22153
I20260812 19:58:08.242161 22223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:08.358866 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 22153
Shutting down hive metastore.
2026-08-12T19:58:08Z chronyd exiting
Aug 12 19:58:08 dist-test-slave-p3t5 krb5kdc[23300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:58:08 dist-test-slave-p3t5 krb5kdc[23300](info): set up 2 sockets
Aug 12 19:58:08 dist-test-slave-p3t5 krb5kdc[23300](info): commencing operation
krb5kdc: starting...
W20260812 19:58:10.879124 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.100s user 0.005s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:58:11 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:58:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:11Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:58:12: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:58:14 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564694, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:19 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564699, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:19 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564699, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:20.937582 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:21.325752 23521 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:21.326198 23521 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:21.326364 23521 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:21.335139 23521 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:21.335312 23521 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:21.335449 23521 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:21.335551 23521 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:21.347004 23521 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:21.348834 23521 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:21.350477 23521 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:21.360105 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:21.360308 23530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:21.360134 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:21.360981 23521 server_base.cc:1061] running on GCE node
I20260812 19:58:21.362016 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:21.363598 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:21.364873 23521 hybrid_clock.cc:648] HybridClock initialized: now 1786564701364825 us; error 46 us; skew 500 ppm
I20260812 19:58:21.365262 23521 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:21 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564701, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:21.369196 23521 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:21.371137 23521 webserver.cc:533] Webserver started at http://127.25.14.62:43059/ using document root <none> and password file <none>
I20260812 19:58:21.372002 23521 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:21.372140 23521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:21.378249 23521 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:58:21.381399 23537 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:21.382956 23521 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:58:21.383165 23521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:21.383953 23521 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:21.413727 23521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:21.421475 23521 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:21.421818 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:21.444458 23521 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35465
I20260812 19:58:21.444490 23597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35465 every 8 connection(s)
I20260812 19:58:21.446452 23521 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:21.454851 23598 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:58:21.456058 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 23521
I20260812 19:58:21.456281 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:58:21.459301 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap starting.
I20260812 19:58:21.480140 23598 log.cc:826] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:21.493466 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:21.494235 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap complete.
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:58:21.502127 23598 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:21.502970 23598 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4307db228843427aa13bef0bf7bfca2d, State: Initialized, Role: FOLLOWER
I20260812 19:58:21.503801 23598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:21.504837 23602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:21.505170 23602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:21.506218 23598 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:21.517575 23615 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:21.517803 23615 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: loading cluster ID for follower catalog manager: success
I20260812 19:58:21.522353 23615 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: acquiring CA information for follower catalog manager: success
I20260812 19:58:21.526024 23615 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:58:21.543390 23521 master_runner.cc:187] Error getting master registration for 127.25.14.61:46865: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46865: connect: Connection refused (error 111),
I20260812 19:58:21.547998 23521 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:21.633410 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46865
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:22.034184 23623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:22.034611 23623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:22.034700 23623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:22.043242 23623 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:22.043360 23623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:22.043463 23623 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:22.043566 23623 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:22.055282 23623 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:22.057174 23623 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:22.058794 23623 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:22.068127 23629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:22.068100 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:22.068946 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:22.070025 23623 server_base.cc:1061] running on GCE node
I20260812 19:58:22.070892 23623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:22.072430 23623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:22.073736 23623 hybrid_clock.cc:648] HybridClock initialized: now 1786564702073679 us; error 39 us; skew 500 ppm
I20260812 19:58:22.074149 23623 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564702, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:22.077994 23623 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:58:22.079783 23623 webserver.cc:533] Webserver started at http://127.25.14.61:39949/ using document root <none> and password file <none>
I20260812 19:58:22.080690 23623 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:22.080866 23623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:22.086935 23623 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:58:22.090197 23640 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:22.091683 23623 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:58:22.091890 23623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "139f449766ad4191bb0595834283db3b"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:22.092707 23623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:22.126211 23623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:22.130843 23623 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:22.131170 23623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:22.153761 23700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46865 every 8 connection(s)
I20260812 19:58:22.153791 23623 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46865
I20260812 19:58:22.155612 23623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:58:22.157182 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 23623
I20260812 19:58:22.157428 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:58:22.166262 23701 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:58:22.171876 23701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Bootstrap starting.
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:58:22.190554 23701 log.cc:826] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:22.203329 23701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:22.204101 23701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Bootstrap complete.
I20260812 19:58:22.211856 23701 raft_consensus.cc:359] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.212638 23701 raft_consensus.cc:740] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139f449766ad4191bb0595834283db3b, State: Initialized, Role: FOLLOWER
I20260812 19:58:22.213490 23701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.214567 23707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:22.214857 23707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:22.215849 23701 sys_catalog.cc:565] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:22.227452 23720 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:22.227698 23720 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: loading cluster ID for follower catalog manager: success
I20260812 19:58:22.233479 23720 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: acquiring CA information for follower catalog manager: success
I20260812 19:58:22.237325 23720 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:22.263614 23623 master_runner.cc:187] Error getting master registration for 127.25.14.60:40063: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40063: connect: Connection refused (error 111),
I20260812 19:58:22.267948 23623 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:22.325363 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40063
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:22.745088 23728 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:22.745491 23728 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:22.745705 23728 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:22.754442 23728 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:22.754565 23728 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:22.754650 23728 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:22.754761 23728 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:22.766819 23728 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:22.768646 23728 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:22.770393 23728 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:22.780112 23735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:22.780113 23734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:22.781011 23737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:22.781683 23728 server_base.cc:1061] running on GCE node
I20260812 19:58:22.782579 23728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:22.784157 23728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:22.785419 23728 hybrid_clock.cc:648] HybridClock initialized: now 1786564702785354 us; error 92 us; skew 500 ppm
I20260812 19:58:22.785902 23728 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564702, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:22.789933 23728 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:58:22.791926 23728 webserver.cc:533] Webserver started at http://127.25.14.60:43971/ using document root <none> and password file <none>
I20260812 19:58:22.792773 23728 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:22.792927 23728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:22.799407 23728 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:22.802779 23745 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:22.804311 23728 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:58:22.804538 23728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "0a43729acacb43ac9a3089b405d536a1"
format_stamp: "Formatted at 2026-08-12 19:58:02 on dist-test-slave-p3t5"
I20260812 19:58:22.805313 23728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:22.840057 23728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:22.844681 23728 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:22.845005 23728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:22.868158 23728 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:40063
I20260812 19:58:22.868194 23803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:40063 every 8 connection(s)
I20260812 19:58:22.870096 23728 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:58:22.872476 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 23728
I20260812 19:58:22.872694 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:58:22.880110 23804 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:22.884117 23804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Bootstrap starting.
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:22.904155 23804 log.cc:826] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:22.914381 23804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:22.914997 23804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: Bootstrap complete.
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:22.921952 23804 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.922922 23804 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a43729acacb43ac9a3089b405d536a1, State: Initialized, Role: FOLLOWER
I20260812 19:58:22.923787 23804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.924841 23812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:22.925176 23812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:22.926254 23804 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:22.941072 23827 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:22.941242 23827 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: loading cluster ID for follower catalog manager: success
I20260812 19:58:22.945850 23827 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: acquiring CA information for follower catalog manager: success
I20260812 19:58:22.949024 23827 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:58:22 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564702, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:22.972760 23830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:58:22.973109 23830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.976178 23830 leader_election.cc:290] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139f449766ad4191bb0595834283db3b (127.25.14.61:46865), 0a43729acacb43ac9a3089b405d536a1 (127.25.14.60:40063)
I20260812 19:58:22.985462 23728 master_runner.cc:264] No leader master found from master 0a43729acacb43ac9a3089b405d536a1
I20260812 19:58:22.985641 23728 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:22.985479 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4307db228843427aa13bef0bf7bfca2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "139f449766ad4191bb0595834283db3b" is_pre_election: true
I20260812 19:58:22.986164 23674 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4307db228843427aa13bef0bf7bfca2d in term 1.
I20260812 19:58:22.986868 23541 leader_election.cc:304] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139f449766ad4191bb0595834283db3b, 4307db228843427aa13bef0bf7bfca2d; no voters:
I20260812 19:58:22.987421 23830 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:58:22.987598 23830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:58:22.987689 23830 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:22.990717 23774 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4307db228843427aa13bef0bf7bfca2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a43729acacb43ac9a3089b405d536a1" is_pre_election: true
I20260812 19:58:22.991441 23774 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4307db228843427aa13bef0bf7bfca2d in term 1.
I20260812 19:58:22.992427 23830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:22.993171 23830 leader_election.cc:290] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [CANDIDATE]: Term 2 election: Requested vote from peers 139f449766ad4191bb0595834283db3b (127.25.14.61:46865), 0a43729acacb43ac9a3089b405d536a1 (127.25.14.60:40063)
I20260812 19:58:22.993690 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4307db228843427aa13bef0bf7bfca2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "139f449766ad4191bb0595834283db3b"
I20260812 19:58:22.993973 23674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:22.994689 23774 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4307db228843427aa13bef0bf7bfca2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a43729acacb43ac9a3089b405d536a1"
I20260812 19:58:22.994907 23774 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:22.998250 23674 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4307db228843427aa13bef0bf7bfca2d in term 2.
I20260812 19:58:23.000141 23541 leader_election.cc:304] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139f449766ad4191bb0595834283db3b, 4307db228843427aa13bef0bf7bfca2d; no voters:
I20260812 19:58:23.000564 23830 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:58:23.001235 23774 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4307db228843427aa13bef0bf7bfca2d in term 2.
I20260812 19:58:23.001492 23830 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 LEADER]: Becoming Leader. State: Replica: 4307db228843427aa13bef0bf7bfca2d, State: Running, Role: LEADER
I20260812 19:58:23.002156 23830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:23.005978 23834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4307db228843427aa13bef0bf7bfca2d. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.006395 23834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:23.007340 23836 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:58:23.009565 23836 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:58:23.010545 23836 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:23.010655 23836 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:58:23.011924 23836 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:58:23.012734 23836 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Loaded TSK: 0
I20260812 19:58:23.013736 23836 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:23.177075 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:23.579005 23774 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 2 FOLLOWER]: Refusing update from remote peer 4307db228843427aa13bef0bf7bfca2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:58:23.581606 23674 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Refusing update from remote peer 4307db228843427aa13bef0bf7bfca2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260812 19:58:23.581501 23842 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:23.581944 23842 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260812 19:58:23.581776 23843 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260812 19:58:23.582087 23842 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:58:23.582731 23849 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:58:23.586612 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4307db228843427aa13bef0bf7bfca2d. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.587013 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:23.590384 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.590632 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:23.594645 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.594905 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:58:23.596175 23842 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:23.596454 23842 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:23.600111 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4307db228843427aa13bef0bf7bfca2d. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.600499 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:23.602181 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.602463 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:23.604858 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4307db228843427aa13bef0bf7bfca2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:23.605111 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:23.611804 23842 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:23.613657 23842 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:23.615109 23842 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:23.627348 23862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:23.627992 23863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:23.631482 23865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:23.632130 23842 server_base.cc:1061] running on GCE node
I20260812 19:58:23.632835 23842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:23.634348 23842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:23.635591 23842 hybrid_clock.cc:648] HybridClock initialized: now 1786564703635529 us; error 56 us; skew 500 ppm
I20260812 19:58:23.636004 23842 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:23 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564703, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:23.641459 23842 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:23.644249 23842 webserver.cc:533] Webserver started at http://127.25.14.1:45137/ using document root <none> and password file <none>
I20260812 19:58:23.645287 23842 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:23.645428 23842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:23.653600 23842 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 19:58:23.658324 23873 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:23.660451 23842 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 19:58:23.660661 23842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:23.661471 23842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:23.679003 23842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:23.683025 23842 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:23.683331 23842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:23.684396 23842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:23.686120 23842 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:23.686254 23842 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:23.686384 23842 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:23.686474 23842 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:23.723503 23842 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33033
I20260812 19:58:23.723526 24003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33033 every 8 connection(s)
I20260812 19:58:23.725337 23842 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:23.727413 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 23842
I20260812 19:58:23.727653 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:58:23 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564703, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:23 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564703, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:23.762279 24005 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:23.762794 24005 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:23.764060 24005 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:23.764961 24006 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:23.765178 24006 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:23.765899 24006 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
Aug 12 19:58:23 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564703, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:23.767151 23762 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
I20260812 19:58:23.767716 23556 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
I20260812 19:58:23.770277 23556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:47841
I20260812 19:58:23.775597 24007 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:23.775795 24007 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:23.776182 24007 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:23.778086 23662 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:23.901248 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:24.290896 24023 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:24.291285 24023 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:24.291478 24023 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:24.300560 24023 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:24.300848 24023 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:58:24.314065 24023 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:24.315902 24023 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:24.317447 24023 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:24.328650 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:24.329373 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:24.330312 24031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:24.330849 24023 server_base.cc:1061] running on GCE node
I20260812 19:58:24.331570 24023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:24.332625 24023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:24.333881 24023 hybrid_clock.cc:648] HybridClock initialized: now 1786564704333799 us; error 76 us; skew 500 ppm
I20260812 19:58:24.334235 24023 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:24 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564704, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:24.338162 24023 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:58:24.340112 24023 webserver.cc:533] Webserver started at http://127.25.14.2:36485/ using document root <none> and password file <none>
I20260812 19:58:24.340951 24023 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:24.341120 24023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:24.347183 24023 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:24.350251 24040 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:24.351847 24023 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:58:24.352058 24023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "1681a5ec2583418fb3fbc4c16989d31d"
format_stamp: "Formatted at 2026-08-12 19:58:04 on dist-test-slave-p3t5"
I20260812 19:58:24.352876 24023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:24.372619 24023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:24.377123 24023 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:24.377460 24023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:24.378701 24023 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:24.380558 24023 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:24.380697 24023 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:24.380856 24023 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:24.380951 24023 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:24.419075 24023 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:39835
I20260812 19:58:24.419134 24168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:39835 every 8 connection(s)
I20260812 19:58:24.420972 24023 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:58:24.430877 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 24023
I20260812 19:58:24.431119 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:58:24 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564704, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:24 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564704, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:24 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564704, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:24.453666 24171 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:24.453743 24170 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:24.454149 24171 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:24.454155 24170 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:24.455236 24171 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:24.455295 24170 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:24.457489 23662 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:39835)
I20260812 19:58:24.458204 23556 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:39835)
I20260812 19:58:24.459583 23556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:57219
I20260812 19:58:24.463806 24169 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:24.463963 24169 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:24.464336 24169 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:24.465582 23762 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:39835)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:24.587263 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:24.774088 24006 heartbeater.cc:499] Master 127.25.14.62:35465 was elected leader, sending a full tablet report...
W20260812 19:58:24.967366 24184 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:24.967733 24184 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:24.967825 24184 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:24.976796 24184 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:24.977005 24184 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:58:24.988396 24184 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:24.990293 24184 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:24.991721 24184 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:25.002727 24192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:25.002761 24189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:25.002693 24190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:25.003676 24184 server_base.cc:1061] running on GCE node
I20260812 19:58:25.004410 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:25.005517 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:25.006783 24184 hybrid_clock.cc:648] HybridClock initialized: now 1786564705006679 us; error 85 us; skew 500 ppm
I20260812 19:58:25.007256 24184 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564705, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:25.011670 24184 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:58:25.013723 24184 webserver.cc:533] Webserver started at http://127.25.14.3:33853/ using document root <none> and password file <none>
I20260812 19:58:25.014631 24184 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:25.014809 24184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:25.021451 24184 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.004s
I20260812 19:58:25.024747 24200 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:25.026415 24184 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:58:25.026636 24184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "b97a2e411090422c99b6a839f1d73f72"
format_stamp: "Formatted at 2026-08-12 19:58:05 on dist-test-slave-p3t5"
I20260812 19:58:25.027536 24184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:25.052165 24184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:25.056617 24184 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:25.056934 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:25.058190 24184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:25.059945 24184 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:25.060042 24184 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:25.060220 24184 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:25.060319 24184 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:25.099063 24184 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:38815
I20260812 19:58:25.099117 24335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:38815 every 8 connection(s)
I20260812 19:58:25.100821 24184 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:58:25.105446 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 24184
I20260812 19:58:25.105724 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564705, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564705, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564705, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:25.144593 24338 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:25.145087 24338 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:25.145558 24336 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:25.145736 24336 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:25.146384 24338 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:25.146407 24336 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:25.148512 24337 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:25.148609 23762 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:38815)
I20260812 19:58:25.148669 24337 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:25.148743 23662 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:38815)
I20260812 19:58:25.149210 24337 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:25.150318 23556 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:38815)
I20260812 19:58:25.151124 23556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:51365
I20260812 19:58:25.164814 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:25.173503 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:25.173717 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23521
I20260812 19:58:25.174728 23593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:25.289700 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 23521
I20260812 19:58:25.322405 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--webserver_interface=127.25.14.62
--webserver_port=43059
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:25.463780 24170 heartbeater.cc:499] Master 127.25.14.62:35465 was elected leader, sending a full tablet report...
W20260812 19:58:25.465747 24170 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111)
W20260812 19:58:25.714346 24349 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:25.714705 24349 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:25.714797 24349 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:25.723305 24349 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:25.723413 24349 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:25.723520 24349 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:25.723595 24349 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:25.735384 24349 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=43059
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:25.737149 24349 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:25.738766 24349 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:25.747838 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:25.747808 24357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:25.748606 24359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:25.749150 24349 server_base.cc:1061] running on GCE node
I20260812 19:58:25.750056 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:25.751509 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:25.752713 24349 hybrid_clock.cc:648] HybridClock initialized: now 1786564705752637 us; error 70 us; skew 500 ppm
I20260812 19:58:25.753099 24349 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564705, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:25.757007 24349 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:25.758929 24349 webserver.cc:533] Webserver started at http://127.25.14.62:43059/ using document root <none> and password file <none>
I20260812 19:58:25.759719 24349 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:25.759868 24349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:25.766036 24349 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:58:25.769567 24369 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:25.771323 24349 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:25.771531 24349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4307db228843427aa13bef0bf7bfca2d"
format_stamp: "Formatted at 2026-08-12 19:58:01 on dist-test-slave-p3t5"
I20260812 19:58:25.772437 24349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 19:58:25.778780 24006 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:35465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111)
I20260812 19:58:25.797897 24349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:25.802558 24349 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:25.802796 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:25.825436 24349 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35465
I20260812 19:58:25.825469 24423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35465 every 8 connection(s)
I20260812 19:58:25.827256 24349 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:25.830443 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 24349
I20260812 19:58:25.830843 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23842
I20260812 19:58:25.836103 24424 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:58:25.839995 24424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap starting.
Aug 12 19:58:25 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:58:25.850746 23997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:25.860630 24424 log.cc:826] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:25.871827 24424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:25.872566 24424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: Bootstrap complete.
I20260812 19:58:25.883277 24424 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:25.884086 24424 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4307db228843427aa13bef0bf7bfca2d, State: Initialized, Role: FOLLOWER
I20260812 19:58:25.884972 24424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:25.886011 24431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:25.886304 24431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:25.887158 24424 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:25.898978 24447 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:25.899237 24447 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: loading cluster ID for follower catalog manager: success
I20260812 19:58:25.905184 24447 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: acquiring CA information for follower catalog manager: success
I20260812 19:58:25.908941 24447 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:58:25.952780 24349 master_runner.cc:264] No leader master found from master 4307db228843427aa13bef0bf7bfca2d
I20260812 19:58:25.952965 24349 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:25.959659 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 23842
I20260812 19:58:25.985371 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33033
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=45137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:26.153306 24337 heartbeater.cc:499] Master 127.25.14.62:35465 was elected leader, sending a full tablet report...
I20260812 19:58:26.178622 24388 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b97a2e411090422c99b6a839f1d73f72" instance_seqno: 1786564705089529) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:33359; Asking this server to re-register.
I20260812 19:58:26.179661 24337 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:26.181521 24388 ts_manager.cc:194] Registered new tserver with Master: b97a2e411090422c99b6a839f1d73f72 (127.25.14.3:38815)
W20260812 19:58:26.381419 24455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:26.381901 24455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:26.382063 24455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:26.391170 24455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:26.391422 24455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:26.403113 24455 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:26.405036 24455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:26.406708 24455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:26.417870 24463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:26.418161 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:26.418186 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:26.422381 24455 server_base.cc:1061] running on GCE node
I20260812 19:58:26.423128 24455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:26.424233 24455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:26.425457 24455 hybrid_clock.cc:648] HybridClock initialized: now 1786564706425382 us; error 61 us; skew 500 ppm
I20260812 19:58:26.425833 24455 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:26 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564706, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:26.429919 24455 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:26.431783 24455 webserver.cc:533] Webserver started at http://127.25.14.1:45137/ using document root <none> and password file <none>
I20260812 19:58:26.432586 24455 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:26.432718 24455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:26.438756 24455 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:26.441533 24473 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:26.443209 24455 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:58:26.443437 24455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:26.444232 24455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:26.463465 24455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:26.473062 24455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:26.473433 24455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:26.473924 24170 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:26.474176 24170 heartbeater.cc:499] Master 127.25.14.62:35465 was elected leader, sending a full tablet report...
I20260812 19:58:26.474867 24455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:26.475260 24388 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1681a5ec2583418fb3fbc4c16989d31d" instance_seqno: 1786564704409882) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:34857; Asking this server to re-register.
I20260812 19:58:26.475919 24170 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:26.477072 24455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:26.477166 24455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:26.477232 24388 ts_manager.cc:194] Registered new tserver with Master: 1681a5ec2583418fb3fbc4c16989d31d (127.25.14.2:39835)
I20260812 19:58:26.477337 24455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:26.477411 24455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:26.515391 24455 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33033
I20260812 19:58:26.515486 24604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33033 every 8 connection(s)
I20260812 19:58:26.517196 24455 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:26.517994 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 24455
Aug 12 19:58:26 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 19:58:26 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564706, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:26 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564706, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:26.559768 24615 raft_consensus.cc:493] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4307db228843427aa13bef0bf7bfca2d)
I20260812 19:58:26.560032 24615 raft_consensus.cc:515] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:26.562697 24615 leader_election.cc:290] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465), 0a43729acacb43ac9a3089b405d536a1 (127.25.14.60:40063)
I20260812 19:58:26.573385 24605 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40063
I20260812 19:58:26.573956 24605 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:26.575414 24605 heartbeater.cc:507] Master 127.25.14.60:40063 requested a full tablet report, sending...
I20260812 19:58:26.578289 23762 ts_manager.cc:194] Re-registered known tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
I20260812 19:58:26.588238 24606 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35465
I20260812 19:58:26.588549 24606 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:26.589260 24606 heartbeater.cc:507] Master 127.25.14.62:35465 requested a full tablet report, sending...
I20260812 19:58:26.590749 24388 ts_manager.cc:194] Registered new tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
I20260812 19:58:26.596201 24399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "139f449766ad4191bb0595834283db3b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4307db228843427aa13bef0bf7bfca2d" is_pre_election: true
I20260812 19:58:26.596849 24399 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139f449766ad4191bb0595834283db3b in term 2.
I20260812 19:58:26.597038 23774 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "139f449766ad4191bb0595834283db3b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a43729acacb43ac9a3089b405d536a1" is_pre_election: true
I20260812 19:58:26.597921 23641 leader_election.cc:304] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139f449766ad4191bb0595834283db3b, 4307db228843427aa13bef0bf7bfca2d; no voters:
I20260812 19:58:26.598446 24615 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:58:26.598557 24615 raft_consensus.cc:493] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4307db228843427aa13bef0bf7bfca2d)
I20260812 19:58:26.598662 24615 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 2 FOLLOWER]: Advancing to term 3
Aug 12 19:58:26 dist-test-slave-p3t5 krb5kdc[23300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564706, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:26.601771 24615 raft_consensus.cc:515] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:26.603114 24615 leader_election.cc:290] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [CANDIDATE]: Term 3 election: Requested vote from peers 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465), 0a43729acacb43ac9a3089b405d536a1 (127.25.14.60:40063)
I20260812 19:58:26.603952 23774 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "139f449766ad4191bb0595834283db3b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a43729acacb43ac9a3089b405d536a1"
I20260812 19:58:26.605263 24399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "139f449766ad4191bb0595834283db3b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4307db228843427aa13bef0bf7bfca2d"
I20260812 19:58:26.605522 24399 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:58:26.610791 24607 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46865
I20260812 19:58:26.611020 24607 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:26.611486 24607 heartbeater.cc:507] Master 127.25.14.61:46865 requested a full tablet report, sending...
I20260812 19:58:26.611495 24399 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139f449766ad4191bb0595834283db3b in term 3.
I20260812 19:58:26.612015 23641 leader_election.cc:304] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 139f449766ad4191bb0595834283db3b, 4307db228843427aa13bef0bf7bfca2d; no voters: 0a43729acacb43ac9a3089b405d536a1
I20260812 19:58:26.612409 24615 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:58:26.613080 24615 raft_consensus.cc:697] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [term 3 LEADER]: Becoming Leader. State: Replica: 139f449766ad4191bb0595834283db3b, State: Running, Role: LEADER
I20260812 19:58:26.613720 24615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:26.616056 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139f449766ad4191bb0595834283db3b. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:26.616400 23662 ts_manager.cc:194] Re-registered known tserver with Master: 7ca7688ddb7c40ffb9158a3bca13f79b (127.25.14.1:33033)
I20260812 19:58:26.616492 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:26.623448 24625 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:58:26.625416 24626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4307db228843427aa13bef0bf7bfca2d)
I20260812 19:58:26.625921 24626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } }
I20260812 19:58:26.626304 24625 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:58:26.627573 24625 catalog_manager.cc:1295] Loaded cluster ID: 370a61f24f2f4f018f9f07416e30e9fd
I20260812 19:58:26.627669 24625 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:58:26.628921 24626 leader_election.cc:290] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4307db228843427aa13bef0bf7bfca2d (127.25.14.62:35465), 139f449766ad4191bb0595834283db3b (127.25.14.61:46865)
I20260812 19:58:26.629352 24625 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:58:26.629336 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "139f449766ad4191bb0595834283db3b" is_pre_election: true
I20260812 19:58:26.630710 24625 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b: Loaded TSK: 0
I20260812 19:58:26.632000 24625 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:58:26.641423 24399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a43729acacb43ac9a3089b405d536a1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4307db228843427aa13bef0bf7bfca2d" is_pre_election: true
I20260812 19:58:26.641919 24399 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a43729acacb43ac9a3089b405d536a1 in current term 3: Already voted for candidate 139f449766ad4191bb0595834283db3b in this term.
I20260812 19:58:26.643064 23746 leader_election.cc:304] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a43729acacb43ac9a3089b405d536a1; no voters: 139f449766ad4191bb0595834283db3b, 4307db228843427aa13bef0bf7bfca2d
I20260812 19:58:26.643605 24626 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:58:26.646162 24626 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:58:26.745379 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24455
W20260812 19:58:26.746624 24632 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:26.742900 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.25.14.1:33033)
0812 19:58:26.743667 (+ 767us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:26.743674 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:26.743708 (+ 34us) server_negotiation.cc:408] Connection header received
0812 19:58:26.743776 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:26.743784 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:26.744006 (+ 222us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:26.744195 (+ 189us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:26.745141 (+ 946us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57466: BlockingRecv error: recv got EOF from 127.0.0.1:57466 (error 108)
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":381,"threads_started":1}
W20260812 19:58:26.748117 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24455: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:33033: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:58:26.748217 24632 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:26.746383 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.25.14.1:33033)
0812 19:58:26.746984 (+ 601us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:26.746989 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:26.747012 (+ 23us) server_negotiation.cc:408] Connection header received
0812 19:58:26.747075 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:26.747081 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:26.747154 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:26.747276 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:26.748032 (+ 756us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57472: BlockingRecv error: recv got EOF from 127.0.0.1:57472 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":773,"threads_started":1}
I20260812 19:58:26.748272 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 24455
I20260812 19:58:26.773108 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33033
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=45137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:27.016247 24399 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [term 3 FOLLOWER]: Refusing update from remote peer 139f449766ad4191bb0595834283db3b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:58:27.017338 24622 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:58:27.022212 24635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139f449766ad4191bb0595834283db3b. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.022615 24635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:27.023857 24615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.024164 24615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:27.024530 23774 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [term 3 FOLLOWER]: Refusing update from remote peer 139f449766ad4191bb0595834283db3b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:58:27.025302 24622 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:58:27.025332 24635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.025560 24635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4307db228843427aa13bef0bf7bfca2d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:27.028211 24642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.028582 24642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:27.028532 24626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139f449766ad4191bb0595834283db3b. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.028802 24626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a43729acacb43ac9a3089b405d536a1 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:58:27.173863 24634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:27.174244 24634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:27.174430 24634 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:58:27.174521 24634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:27.183741 24634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:27.184044 24634 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:27.195772 24634 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:27.197714 24634 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:27.199301 24634 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:27.211324 24647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:27.211341 24648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:27.215102 24650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:27.215831 24634 server_base.cc:1061] running on GCE node
I20260812 19:58:27.216665 24634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:27.217921 24634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:27.219233 24634 hybrid_clock.cc:648] HybridClock initialized: now 1786564707219159 us; error 40 us; skew 500 ppm
I20260812 19:58:27.219589 24634 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:27 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564707, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:27.223677 24634 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:27.225512 24634 webserver.cc:533] Webserver started at http://127.25.14.1:45137/ using document root <none> and password file <none>
I20260812 19:58:27.226387 24634 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:27.226480 24634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:27.232581 24634 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:58:27.235598 24657 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:27.237059 24634 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:58:27.237211 24634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:27.238006 24634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:27.256524 24634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:27.261096 24634 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:27.261368 24634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:27.262486 24634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:27.264213 24634 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:27.264297 24634 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:27.264385 24634 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:27.264441 24634 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:58:27.264781 24634 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:58:27.264842 24634 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:58:27.264950 24634 fault_injection.cc:59] injecting fault for kudu (pid 24634): FLAGS_fault_before_start (process will exit)
I20260812 19:58:27.290818 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33033
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--webserver_port=45137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:27.519519 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "139f449766ad4191bb0595834283db3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4307db228843427aa13bef0bf7bfca2d" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35465 } } peers { permanent_uuid: "139f449766ad4191bb0595834283db3b" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46865 } } peers { permanent_uuid: "0a43729acacb43ac9a3089b405d536a1" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40063 } } }
I20260812 19:58:27.519824 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139f449766ad4191bb0595834283db3b [sys.catalog]: This master's current role is: LEADER
W20260812 19:58:27.676539 24666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:27.676904 24666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:27.677085 24666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:27.686102 24666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:27.686359 24666 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:27.697469 24666 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35465,127.25.14.61:46865,127.25.14.60:40063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:27.699252 24666 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:27.700757 24666 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:27.711462 24675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:27.711521 24678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:27.711653 24674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:27.713301 24666 server_base.cc:1061] running on GCE node
I20260812 19:58:27.714200 24666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:27.715273 24666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:27.716538 24666 hybrid_clock.cc:648] HybridClock initialized: now 1786564707716497 us; error 46 us; skew 500 ppm
I20260812 19:58:27.716953 24666 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:27 dist-test-slave-p3t5 krb5kdc[23300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564707, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:27.721091 24666 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:27.722899 24666 webserver.cc:533] Webserver started at http://127.25.14.1:45137/ using document root <none> and password file <none>
I20260812 19:58:27.723652 24666 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:27.723768 24666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:27.729888 24666 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:58:27.732893 24687 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:27.734584 24666 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:58:27.734756 24666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7ca7688ddb7c40ffb9158a3bca13f79b"
format_stamp: "Formatted at 2026-08-12 19:58:03 on dist-test-slave-p3t5"
I20260812 19:58:27.735466 24666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:27.759805 24666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:27.764118 24666 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:27.764379 24666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:27.765414 24666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:27.767107 24666 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:27.767196 24666 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:27.767279 24666 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:27.767339 24666 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:27.804700 24666 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33033
I20260812 19:58:27.804759 24812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33033 every 8 connection(s)
I20260812 19:58:27.806562 24666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:27.809463 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 24666
I20260812 19:58:27.809834 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24666
W20260812 19:58:27.831800 24822 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:27.811704 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.25.14.1:33033)
0812 19:58:27.812600 (+ 896us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:27.812622 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:27.827159 (+ 14537us) server_negotiation.cc:408] Connection header received
0812 19:58:27.828137 (+ 978us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:27.828150 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:27.828448 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:27.828706 (+ 258us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:27.830220 (+ 1514us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57478: BlockingRecv error: recv got EOF from 127.0.0.1:57478 (error 108)
Metrics: {"server-negotiator.queue_time_us":700,"spinlock_wait_cycles":71936,"thread_start_us":543,"threads_started":1}
W20260812 19:58:27.832379 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24666: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:33033: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:27.832563 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 24666
I20260812 19:58:27.857760 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24023
W20260812 19:58:27.858618 24821 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:27.811244 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:53921 (local address 127.25.14.60:40063)
0812 19:58:27.827382 (+ 16138us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:27.827386 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:27.827402 (+ 16us) server_negotiation.cc:408] Connection header received
0812 19:58:27.827457 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:27.827462 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:27.827524 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:27.827672 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:27.830212 (+ 2540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:27.831607 (+ 1395us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:27.857509 (+ 25902us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:53921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":16037,"thread_start_us":11597,"threads_started":1}
W20260812 19:58:27.858637 24820 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:27.810915 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:39989 (local address 127.25.14.62:35465)
0812 19:58:27.832653 (+ 21738us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:27.832663 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:27.836328 (+ 3665us) server_negotiation.cc:408] Connection header received
0812 19:58:27.836423 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:27.836433 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:27.836547 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:27.836679 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:27.857512 (+ 20833us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:39989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":21472,"thread_start_us":21351,"threads_started":1}
W20260812 19:58:27.859251 24825 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:27.818773 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:46231 (local address 127.25.14.61:46865)
0812 19:58:27.820772 (+ 1999us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:27.820779 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:27.822550 (+ 1771us) server_negotiation.cc:408] Connection header received
0812 19:58:27.822717 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:27.822728 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:27.822847 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:27.823007 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:27.832498 (+ 9491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:27.838137 (+ 5639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:27.857471 (+ 19334us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:46231: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":1866,"thread_start_us":1757,"threads_started":1}
W20260812 19:58:27.860999 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24023: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:39835: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:27.861202 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 24023
I20260812 19:58:27.883595 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24184
W20260812 19:58:27.887145 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24184: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:38815: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:27.887347 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 24184
I20260812 19:58:27.912484 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24349
W20260812 19:58:27.914953 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 24349: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:35465: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:27.915170 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 24349
W20260812 19:58:27.915177 24820 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:27.913678 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.25.14.62:35465)
0812 19:58:27.913878 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:27.913883 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:27.913903 (+ 20us) server_negotiation.cc:408] Connection header received
0812 19:58:27.913971 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:27.913977 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:27.914050 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:27.914150 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:27.914932 (+ 782us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43118: BlockingRecv error: recv got EOF from 127.0.0.1:43118 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 19:58:27.944892 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23623
I20260812 19:58:27.946307 23696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:27.987628 23728 master_runner.cc:187] Error getting master registration for 127.25.14.62:35465: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35465: connect: Connection refused (error 111),
I20260812 19:58:27.991501 23728 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:28.062918 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 23623
I20260812 19:58:28.093075 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 23728
I20260812 19:58:28.094501 23798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:28.207492 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 23728
Shutting down hive metastore.
2026-08-12T19:58:28Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/10 (47690 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/11
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 19:58:28 dist-test-slave-p3t5 krb5kdc[24842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:58:28 dist-test-slave-p3t5 krb5kdc[24842](info): set up 2 sockets
Aug 12 19:58:28 dist-test-slave-p3t5 krb5kdc[24842](info): commencing operation
krb5kdc: starting...
W20260812 19:58:30.881364 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.238s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:58:31 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:31Z Disabled control of system clock
2026-08-12T19:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:31Z Disabled control of system clock
2026-08-12T19:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:31Z Disabled control of system clock
2026-08-12T19:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:31Z Disabled control of system clock
2026-08-12T19:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:31Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:58:39: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:58:41 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564721, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:48 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:58:48 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:48.501384 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:48.887916 25187 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:48.888350 25187 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:48.888510 25187 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:48.897233 25187 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:48.897403 25187 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:48.897497 25187 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:48.897583 25187 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:48.908888 25187 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:48.910759 25187 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:48.912287 25187 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:48.921871 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:48.921943 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:48.921962 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:48.922787 25187 server_base.cc:1061] running on GCE node
I20260812 19:58:48.923652 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:48.925463 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:48.926766 25187 hybrid_clock.cc:648] HybridClock initialized: now 1786564728926622 us; error 111 us; skew 500 ppm
I20260812 19:58:48.927162 25187 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:48 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:48.930538 25187 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:48.932286 25187 webserver.cc:533] Webserver started at http://127.25.14.62:38899/ using document root <none> and password file <none>
I20260812 19:58:48.933074 25187 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:48.933182 25187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:48.933542 25187 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:48.936244 25187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:58:48.937062 25187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:58:48.942967 25187 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:58:48.947021 25208 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:48.948523 25187 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:58:48.948731 25187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:58:48.948941 25187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:48.965996 25187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:48.973284 25187 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:48.973665 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:48.995533 25187 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:46127
I20260812 19:58:48.995571 25265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:46127 every 8 connection(s)
I20260812 19:58:48.997272 25187 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:48.999768 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25187
I20260812 19:58:49.000001 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:58:49.002226 25267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:49.008361 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:49 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:49.033170 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
W20260812 19:58:49.034765 25211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
W20260812 19:58:49.036590 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:49.086005 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } attempt: 1
W20260812 19:58:49.088521 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:49.154374 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45463
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:49.156836 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } attempt: 2
W20260812 19:58:49.159403 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
I20260812 19:58:49.250746 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } attempt: 3
W20260812 19:58:49.253033 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
I20260812 19:58:49.396415 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } attempt: 4
W20260812 19:58:49.398913 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111)
W20260812 19:58:49.534092 25278 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:49.534475 25278 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:49.534564 25278 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:49.543234 25278 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:49.543361 25278 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:49.543419 25278 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:49.543467 25278 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:49.555274 25278 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:49.557015 25278 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:49.558614 25278 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:49.568006 25285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:49.567984 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:49.568249 25287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:49.569507 25278 server_base.cc:1061] running on GCE node
I20260812 19:58:49.570425 25278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:49.572321 25278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:49.573717 25278 hybrid_clock.cc:648] HybridClock initialized: now 1786564729573559 us; error 113 us; skew 500 ppm
I20260812 19:58:49.574157 25278 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:49 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564729, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:49.577564 25278 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:58:49.579447 25278 webserver.cc:533] Webserver started at http://127.25.14.61:36289/ using document root <none> and password file <none>
I20260812 19:58:49.580235 25278 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:49.580338 25278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:49.580667 25278 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:49.583277 25278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "99459beb0fb9499e83b30f3c0e1f4854"
format_stamp: "Formatted at 2026-08-12 19:58:49 on dist-test-slave-p3t5"
I20260812 19:58:49.584086 25278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "99459beb0fb9499e83b30f3c0e1f4854"
format_stamp: "Formatted at 2026-08-12 19:58:49 on dist-test-slave-p3t5"
I20260812 19:58:49.589900 25278 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 19:58:49.593868 25296 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:49.595551 25278 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:49.595832 25278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "99459beb0fb9499e83b30f3c0e1f4854"
format_stamp: "Formatted at 2026-08-12 19:58:49 on dist-test-slave-p3t5"
I20260812 19:58:49.596098 25278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:49.613508 25278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:49.618014 25278 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:49.618347 25278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:49.640990 25278 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:45463
I20260812 19:58:49.641022 25351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:45463 every 8 connection(s)
I20260812 19:58:49.642823 25278 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:58:49.643226 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25278
I20260812 19:58:49.643461 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:58:49.648155 25352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:49.654764 25352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:49 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564729, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:49.682787 25352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 19:58:49 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564729, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:49.698237 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } attempt: 5
I20260812 19:58:49.702787 25352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:58:49.704286 25298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
Aug 12 19:58:49 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
W20260812 19:58:49.709301 25352 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:49.717608 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:58:49.719694 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:49.758709 25352 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 1
W20260812 19:58:49.761296 25352 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 19:58:49.770947 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 1
W20260812 19:58:49.773455 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:49.809934 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41211
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:49.829701 25352 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 2
W20260812 19:58:49.832192 25352 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:49.841734 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 2
W20260812 19:58:49.844048 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:49.923460 25352 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 3
W20260812 19:58:49.925751 25352 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:49.950333 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 3
W20260812 19:58:49.952692 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:50.069087 25352 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 4
W20260812 19:58:50.071575 25352 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
I20260812 19:58:50.129969 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 4
W20260812 19:58:50.132357 25267 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111)
W20260812 19:58:50.197106 25367 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:50.197492 25367 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:50.197599 25367 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:50.206241 25367 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:50.206367 25367 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:50.206426 25367 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:50.206476 25367 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:50.218341 25367 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:50.220083 25367 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:50.221537 25367 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:50.230783 25372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:50.230863 25375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:50.230829 25373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:50.232292 25367 server_base.cc:1061] running on GCE node
I20260812 19:58:50.233157 25367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:50.234975 25367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:50.236356 25367 hybrid_clock.cc:648] HybridClock initialized: now 1786564730236182 us; error 134 us; skew 500 ppm
I20260812 19:58:50.236778 25367 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:50.240417 25367 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:58:50.242396 25367 webserver.cc:533] Webserver started at http://127.25.14.60:44077/ using document root <none> and password file <none>
I20260812 19:58:50.243238 25367 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:50.243396 25367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:50.243825 25367 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:50.246600 25367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "d0fb7276976e40d799f5fefcc45d6a56"
format_stamp: "Formatted at 2026-08-12 19:58:50 on dist-test-slave-p3t5"
I20260812 19:58:50.247586 25367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "d0fb7276976e40d799f5fefcc45d6a56"
format_stamp: "Formatted at 2026-08-12 19:58:50 on dist-test-slave-p3t5"
I20260812 19:58:50.254096 25367 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 19:58:50.258245 25383 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:50.259974 25367 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:58:50.260238 25367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "d0fb7276976e40d799f5fefcc45d6a56"
format_stamp: "Formatted at 2026-08-12 19:58:50 on dist-test-slave-p3t5"
I20260812 19:58:50.260495 25367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:50.276467 25367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:50.281072 25367 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:50.281419 25367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:50.303679 25367 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:41211
I20260812 19:58:50.303714 25444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:41211 every 8 connection(s)
I20260812 19:58:50.305419 25367 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:58:50.309794 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25367
I20260812 19:58:50.310051 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:58:50.310684 25445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:58:50.320390 25445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:50.350966 25445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:50.365746 25445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:58:50.370874 25352 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 5
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564729, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:50.387636 25445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Bootstrap starting.
I20260812 19:58:50.392079 25445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:50.392949 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Bootstrap starting.
I20260812 19:58:50.393590 25445 log.cc:826] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:50.397050 25445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: No bootstrap required, opened a new log
I20260812 19:58:50.397154 25352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:50.398586 25352 log.cc:826] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:50.401865 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: No bootstrap required, opened a new log
I20260812 19:58:50.404989 25445 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.405413 25445 raft_consensus.cc:385] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:50.405555 25445 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb7276976e40d799f5fefcc45d6a56, State: Initialized, Role: FOLLOWER
I20260812 19:58:50.406389 25445 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.407438 25458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.407728 25458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.408896 25445 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:50.409670 25267 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } attempt: 5
I20260812 19:58:50.409637 25352 raft_consensus.cc:359] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.410001 25352 raft_consensus.cc:385] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:50.410140 25352 raft_consensus.cc:740] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99459beb0fb9499e83b30f3c0e1f4854, State: Initialized, Role: FOLLOWER
I20260812 19:58:50.411017 25352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.415439 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.416188 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.417315 25352 sys_catalog.cc:565] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 19:58:50 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:50.433298 25267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap starting.
I20260812 19:58:50.434577 25477 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 19:58:50.438040 25267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Neither blocks nor log segments found. Creating new log.
I20260812 19:58:50.443390 25267 log.cc:826] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:50.437669 25367 master_runner.cc:187] Error getting master registration for 127.25.14.62:46127: OK, instance_id { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" instance_seqno: 1786564728989496 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:58:50.444024 25367 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:50.448258 25267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: No bootstrap required, opened a new log
I20260812 19:58:50.441291 25278 master_runner.cc:187] Error getting master registration for 127.25.14.62:46127: OK, instance_id { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" instance_seqno: 1786564728989496 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:58:50.450917 25278 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:58:50.451403 25485 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:58:50.451551 25485 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:58:50.456576 25267 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.456890 25267 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:58:50.457062 25267 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Initialized, Role: FOLLOWER
I20260812 19:58:50.457835 25267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.458906 25488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.459239 25488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.460018 25267 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:58:50.477049 25215 tablet.cc:2406] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:58:50.481642 25187 master_runner.cc:264] No leader master found from master 4b71e3aac390423aa4f17e84439ded94
I20260812 19:58:50.481817 25187 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:58:50.488297 25503 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:58:50.488483 25503 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 19:58:50.534060 25389 tablet.cc:2406] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 19:58:50.621466 25302 tablet.cc:2406] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:58:50.724447 25488 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:58:50.724807 25488 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.727010 25488 leader_election.cc:290] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:58:50.728101 25414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56" is_pre_election: true
I20260812 19:58:50.728078 25325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854" is_pre_election: true
I20260812 19:58:50.728646 25414 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 0.
I20260812 19:58:50.728650 25325 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 0.
I20260812 19:58:50.729592 25211 leader_election.cc:304] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters:
I20260812 19:58:50.730142 25488 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:58:50.730419 25488 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:58:50.730549 25488 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:50.734400 25488 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.735319 25488 leader_election.cc:290] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 1 election: Requested vote from peers 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:58:50.736080 25414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56"
I20260812 19:58:50.736090 25325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854"
I20260812 19:58:50.736310 25414 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:50.736337 25325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:58:50.740048 25414 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 1.
I20260812 19:58:50.740072 25325 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 1.
I20260812 19:58:50.741022 25211 leader_election.cc:304] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters:
I20260812 19:58:50.742760 25488 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:58:50.744156 25488 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 LEADER]: Becoming Leader. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Running, Role: LEADER
I20260812 19:58:50.744822 25488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:50.750777 25506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.751159 25506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:50.754315 25509 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:58:50.757041 25509 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:58:50.767140 25325 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 1 FOLLOWER]: Refusing update from remote peer 4b71e3aac390423aa4f17e84439ded94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:58:50.767140 25414 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Refusing update from remote peer 4b71e3aac390423aa4f17e84439ded94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:58:50.768368 25506 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:58:50.768707 25488 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:58:50.796077 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.796399 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.802068 25514 mvcc.cc:204] Tried to move back new op lower bound from 7317769137209335808 to 7317769137136152576. Current Snapshot: MvccSnapshot[applied={T|T < 7317769137209335808}]
I20260812 19:58:50.804708 25458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.805084 25458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.810889 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.811172 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.814440 25506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.814728 25506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:50.817066 25509 catalog_manager.cc:1383] Generated new cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:58:50.817183 25509 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:58:50.819171 25488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.819481 25488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:50.823508 25511 mvcc.cc:204] Tried to move back new op lower bound from 7317769137209335808 to 7317769137136152576. Current Snapshot: MvccSnapshot[applied={T|T < 7317769137209335808}]
I20260812 19:58:50.826550 25458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:50.826825 25458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:50.886920 25509 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:58:50.888871 25509 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:58:50.906445 25509 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Generated new TSK 0
I20260812 19:58:50.907841 25509 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:51.097369 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:51.440935 25477 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:58:51.441067 25477 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: loading cluster ID for follower catalog manager: success
I20260812 19:58:51.445299 25477 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: acquiring CA information for follower catalog manager: success
I20260812 19:58:51.448232 25477 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:58:51.454010 25485 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:58:51.454129 25485 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: loading cluster ID for follower catalog manager: success
I20260812 19:58:51.458055 25485 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: acquiring CA information for follower catalog manager: success
I20260812 19:58:51.460875 25485 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:58:51.485275 25527 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:51.485718 25527 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:51.485888 25527 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:51.494836 25527 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:51.495110 25527 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:51.506843 25527 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:51.508656 25527 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:51.510283 25527 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:51.521451 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:51.521478 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:51.523501 25540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:51.523574 25527 server_base.cc:1061] running on GCE node
I20260812 19:58:51.524400 25527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:51.525823 25527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:51.529187 25527 hybrid_clock.cc:648] HybridClock initialized: now 1786564731528987 us; error 161 us; skew 500 ppm
I20260812 19:58:51.529583 25527 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:51 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:51.533519 25527 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:51.535567 25527 webserver.cc:533] Webserver started at http://127.25.14.1:39811/ using document root <none> and password file <none>
I20260812 19:58:51.536429 25527 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:51.536886 25527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:51.537310 25527 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:51.540001 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:51.540866 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:51.547055 25527 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:58:51.551666 25547 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:51.553941 25527 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:58:51.554183 25527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:51.554427 25527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:51.609766 25527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:51.614403 25527 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:51.614727 25527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:51.615947 25527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:51.617772 25527 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:51.617877 25527 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:51.618036 25527 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:51.618112 25527 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:51.669591 25527 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39037
I20260812 19:58:51.669646 25690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39037 every 8 connection(s)
I20260812 19:58:51.671284 25527 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:51.677021 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25527
I20260812 19:58:51.677291 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:58:51 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:51 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:51 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:51.719267 25692 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:58:51.719812 25692 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:51.721108 25692 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:58:51.724170 25227 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
I20260812 19:58:51.725700 25694 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:58:51.725862 25694 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:51.726429 25694 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:58:51.727896 25227 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:55033
I20260812 19:58:51.728407 25313 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
I20260812 19:58:51.731377 25691 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:58:51.731535 25691 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:51.732161 25691 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:58:51.734282 25402 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:51.863597 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:58:52.257396 25708 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:52.257884 25708 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:52.258064 25708 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:52.267350 25708 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:52.267642 25708 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:58:52.279488 25708 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:52.281380 25708 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:52.282967 25708 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:52.294268 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:52.294348 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:52.295368 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:52.295737 25708 server_base.cc:1061] running on GCE node
I20260812 19:58:52.296439 25708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:52.297788 25708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:52.299046 25708 hybrid_clock.cc:648] HybridClock initialized: now 1786564732298902 us; error 108 us; skew 500 ppm
I20260812 19:58:52.299428 25708 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:52 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564732, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:52.303326 25708 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:58:52.305363 25708 webserver.cc:533] Webserver started at http://127.25.14.2:42401/ using document root <none> and password file <none>
I20260812 19:58:52.306263 25708 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:52.306402 25708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:52.306816 25708 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:52.309489 25708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b"
format_stamp: "Formatted at 2026-08-12 19:58:52 on dist-test-slave-p3t5"
I20260812 19:58:52.310619 25708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b"
format_stamp: "Formatted at 2026-08-12 19:58:52 on dist-test-slave-p3t5"
I20260812 19:58:52.316944 25708 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:58:52.321177 25727 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:52.322819 25708 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:58:52.323029 25708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b"
format_stamp: "Formatted at 2026-08-12 19:58:52 on dist-test-slave-p3t5"
I20260812 19:58:52.323225 25708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:52.341431 25708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:52.345927 25708 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:52.346206 25708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:52.347268 25708 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:52.348932 25708 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:52.349010 25708 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:52.349103 25708 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:52.349148 25708 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:52.388000 25708 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:46491
I20260812 19:58:52.388104 25862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:46491 every 8 connection(s)
I20260812 19:58:52.389714 25708 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:58:52.392254 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25708
I20260812 19:58:52.392513 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:58:52 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564732, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:58:52.428524 25863 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:58:52.428977 25863 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:52.430254 25863 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:58:52.432544 25402 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:46491)
Aug 12 19:58:52 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564732, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:52 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564732, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:52.444438 25865 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:58:52.444648 25865 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:52.445258 25865 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:58:52.446664 25227 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:46491)
I20260812 19:58:52.447988 25227 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:55571
I20260812 19:58:52.448601 25866 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:58:52.448793 25866 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:52.449296 25866 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:58:52.450680 25313 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:46491)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 19:58:52.504241 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.3: real 0.103s user 0.003s sys 0.032s
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:58:52.586030 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:52.734055 25692 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
W20260812 19:58:52.978000 25879 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:52.978403 25879 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:52.978566 25879 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:52.987563 25879 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:52.987835 25879 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:58:52.999221 25879 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:53.001039 25879 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:53.002799 25879 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:53.014191 25887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:53.014222 25888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:53.015102 25891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:53.015637 25879 server_base.cc:1061] running on GCE node
I20260812 19:58:53.016376 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:53.017681 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:53.021032 25879 hybrid_clock.cc:648] HybridClock initialized: now 1786564733020848 us; error 145 us; skew 500 ppm
I20260812 19:58:53.021423 25879 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:53.025256 25879 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:58:53.027200 25879 webserver.cc:533] Webserver started at http://127.25.14.3:36857/ using document root <none> and password file <none>
I20260812 19:58:53.028052 25879 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:53.028209 25879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:53.028679 25879 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:58:53.031666 25879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "baa124c27c1249b59575d27413852084"
format_stamp: "Formatted at 2026-08-12 19:58:53 on dist-test-slave-p3t5"
I20260812 19:58:53.032496 25879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "baa124c27c1249b59575d27413852084"
format_stamp: "Formatted at 2026-08-12 19:58:53 on dist-test-slave-p3t5"
I20260812 19:58:53.038857 25879 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260812 19:58:53.043049 25899 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:53.044715 25879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 19:58:53.044921 25879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "baa124c27c1249b59575d27413852084"
format_stamp: "Formatted at 2026-08-12 19:58:53 on dist-test-slave-p3t5"
I20260812 19:58:53.045161 25879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:53.078426 25879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:53.082965 25879 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:53.083310 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:53.084501 25879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:53.086333 25879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:53.086410 25879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:53.086563 25879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:53.086619 25879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:53.127841 25879 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:32999
I20260812 19:58:53.127892 26027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:32999 every 8 connection(s)
I20260812 19:58:53.129664 25879 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:58:53.135614 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 25879
I20260812 19:58:53.135900 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:58:53.159929 26030 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:58:53.160283 26030 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:53.161235 26030 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:58:53.161794 26029 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:58:53.162048 26029 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:53.162571 26029 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:58:53.163453 26028 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:58:53.163614 26028 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:53.163818 25227 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:32999)
I20260812 19:58:53.163969 25313 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:32999)
I20260812 19:58:53.164098 26028 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:58:53.164813 25227 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:59587
I20260812 19:58:53.166108 25402 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:32999)
I20260812 19:58:53.181006 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:53.190224 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:58:53.190439 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25187
I20260812 19:58:53.191522 25259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:53.308218 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25187
I20260812 19:58:53.344288 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--webserver_interface=127.25.14.62
--webserver_port=38899
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:53.452994 25865 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
W20260812 19:58:53.456136 25865 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:46127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:46127: connect: Connection refused (error 111)
W20260812 19:58:53.738109 25692 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:46127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:46127: connect: Connection refused (error 111)
W20260812 19:58:53.741839 26039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:53.742189 26039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:53.742293 26039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:53.751130 26039 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:58:53.751241 26039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:53.751297 26039 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:58:53.751346 26039 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:58:53.763337 26039 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=38899
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:53.765064 26039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:53.766711 26039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:53.776661 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:53.776609 26046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:53.777066 26047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:53.777799 26039 server_base.cc:1061] running on GCE node
I20260812 19:58:53.778709 26039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:53.780611 26039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:53.782088 26039 hybrid_clock.cc:648] HybridClock initialized: now 1786564733781926 us; error 100 us; skew 500 ppm
I20260812 19:58:53.782514 26039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:53.786303 26039 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:58:53.788300 26039 webserver.cc:533] Webserver started at http://127.25.14.62:38899/ using document root <none> and password file <none>
I20260812 19:58:53.789145 26039 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:53.789275 26039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:53.795816 26039 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:58:53.799252 26058 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:53.800879 26039 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:58:53.801060 26039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:58:53.802135 26039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:53.820609 26039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:53.825454 26039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:53.825809 26039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:53.849336 26039 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:46127
I20260812 19:58:53.849355 26118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:46127 every 8 connection(s)
I20260812 19:58:53.851297 26039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:58:53.853484 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26039
I20260812 19:58:53.853883 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25527
I20260812 19:58:53.860467 26119 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:58:53.864444 26119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap starting.
Aug 12 19:58:53 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:58:53.873701 25686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:53.887848 26119 log.cc:826] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Log is configured to *not* fsync() on all Append() calls
I20260812 19:58:53.900631 26119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:58:53.901368 26119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap complete.
I20260812 19:58:53.912268 26119 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:53.913329 26119 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Initialized, Role: FOLLOWER
I20260812 19:58:53.914395 26119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:53.915364 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:53.915688 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:53.916829 26119 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:58:53.928939 26138 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:58:53.929447 26138 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: loading cluster ID for follower catalog manager: success
I20260812 19:58:53.933840 26138 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: acquiring CA information for follower catalog manager: success
I20260812 19:58:53.936893 26138 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:58:53.975189 26039 master_runner.cc:264] No leader master found from master 4b71e3aac390423aa4f17e84439ded94
I20260812 19:58:53.975342 26039 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:58:53.993513 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25527
I20260812 19:58:54.021109 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39037
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=39811
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:54.167800 26029 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
I20260812 19:58:54.185899 26080 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "baa124c27c1249b59575d27413852084" instance_seqno: 1786564733118581) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:33715; Asking this server to re-register.
I20260812 19:58:54.187281 26029 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:54.189273 26080 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:32999)
I20260812 19:58:54.395792 26149 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:58:54.396023 26149 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:54.397578 26149 leader_election.cc:290] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463)
I20260812 19:58:54.398460 25325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854" is_pre_election: true
I20260812 19:58:54.405848 26091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94" is_pre_election: true
I20260812 19:58:54.406477 26091 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb7276976e40d799f5fefcc45d6a56 in term 1.
I20260812 19:58:54.407168 25387 leader_election.cc:304] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, d0fb7276976e40d799f5fefcc45d6a56; no voters: 99459beb0fb9499e83b30f3c0e1f4854
I20260812 19:58:54.407644 26149 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:58:54.407829 26149 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:58:54.407941 26149 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:54.411630 26149 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:54.412547 26149 leader_election.cc:290] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 2 election: Requested vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463)
I20260812 19:58:54.413187 26091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94"
I20260812 19:58:54.413226 25325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854"
I20260812 19:58:54.413445 26091 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:54.417824 26091 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb7276976e40d799f5fefcc45d6a56 in term 2.
I20260812 19:58:54.418442 25387 leader_election.cc:304] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, d0fb7276976e40d799f5fefcc45d6a56; no voters: 99459beb0fb9499e83b30f3c0e1f4854
I20260812 19:58:54.418846 26149 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:58:54.419389 26149 raft_consensus.cc:697] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 LEADER]: Becoming Leader. State: Replica: d0fb7276976e40d799f5fefcc45d6a56, State: Running, Role: LEADER
I20260812 19:58:54.420011 26149 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:54.422509 26152 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0fb7276976e40d799f5fefcc45d6a56. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.422876 26152 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:54.423743 26154 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:58:54.426509 26154 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:58:54.427616 26154 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:58:54.427721 26154 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:58:54.429065 26154 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:58:54.429977 26154 catalog_manager.cc:6103] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Loaded TSK: 0
I20260812 19:58:54.430872 26154 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260812 19:58:54.454777 26145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:54.455163 26145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:54.455350 26145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:58:54.465243 25865 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:58:54.465551 25865 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
I20260812 19:58:54.466688 26080 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b" instance_seqno: 1786564732378702) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:33239; Asking this server to re-register.
W20260812 19:58:54.467208 26145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:54.467481 26145 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:54.467491 25865 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:54.468820 26080 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:46491)
I20260812 19:58:54.479333 26145 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39811
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:54.481129 26145 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:54.482702 26145 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:54.494572 26161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:54.494654 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:54.494661 26163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:54.496080 26145 server_base.cc:1061] running on GCE node
I20260812 19:58:54.496825 26145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:54.498139 26145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:54.499418 26145 hybrid_clock.cc:648] HybridClock initialized: now 1786564734499255 us; error 124 us; skew 500 ppm
I20260812 19:58:54.499858 26145 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:54 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:54.503603 26145 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:54.505571 26145 webserver.cc:533] Webserver started at http://127.25.14.1:39811/ using document root <none> and password file <none>
I20260812 19:58:54.506433 26145 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:54.506595 26145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:54.513417 26145 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 19:58:54.516882 26174 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:54.518833 26145 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:58:54.519057 26145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:54.519899 26145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:54.539136 26145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:54.543681 26145 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:54.544078 26145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:54.545199 26145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:54.547245 26145 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:54.547382 26145 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:54.547533 26145 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:54.547637 26145 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:54.588959 26145 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39037
I20260812 19:58:54.589072 26315 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39037 every 8 connection(s)
I20260812 19:58:54.590670 26145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:54.600809 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26145
Aug 12 19:58:54 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 19:58:54 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:58:54 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:58:54 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:58:54.624298 26316 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:58:54.624850 26316 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:54.625844 26320 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:58:54.626035 26320 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:54.626140 26316 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:58:54.626636 26320 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:58:54.626766 26318 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:58:54.626924 26318 heartbeater.cc:461] Registering TS with master...
I20260812 19:58:54.627547 26318 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:58:54.628715 25402 ts_manager.cc:194] Re-registered known tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
I20260812 19:58:54.628785 26080 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
I20260812 19:58:54.629014 25313 ts_manager.cc:194] Re-registered known tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:39037)
I20260812 19:58:54.630543 25402 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:50715
I20260812 19:58:54.690742 26329 raft_consensus.cc:493] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:58:54.691002 26329 raft_consensus.cc:515] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:58:54.693395 25414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56" is_pre_election: true
I20260812 19:58:54.694749 26329 leader_election.cc:290] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:58:54.705299 26091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94" is_pre_election: true
I20260812 19:58:54.705823 26091 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99459beb0fb9499e83b30f3c0e1f4854 in current term 2: Already voted for candidate d0fb7276976e40d799f5fefcc45d6a56 in this term.
I20260812 19:58:54.707211 25300 leader_election.cc:304] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99459beb0fb9499e83b30f3c0e1f4854; no voters: 4b71e3aac390423aa4f17e84439ded94, d0fb7276976e40d799f5fefcc45d6a56
I20260812 19:58:54.707844 26329 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:58:54.711172 26329 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:58:54.791903 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26145
W20260812 19:58:54.793195 26336 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:54.788794 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.25.14.1:39037)
0812 19:58:54.789870 (+ 1076us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:54.789878 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:54.789910 (+ 32us) server_negotiation.cc:408] Connection header received
0812 19:58:54.789988 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:54.789996 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:54.790221 (+ 225us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:54.790458 (+ 237us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:54.791598 (+ 1140us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48374: BlockingRecv error: recv got EOF from 127.0.0.1:48374 (error 108)
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":614,"threads_started":1}
W20260812 19:58:54.794880 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26145: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:39037: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:54.795063 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26145
I20260812 19:58:54.822081 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39037
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=39811
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:54.844604 25325 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Refusing update from remote peer d0fb7276976e40d799f5fefcc45d6a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:58:54.845520 26152 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:58:54.847782 26329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0fb7276976e40d799f5fefcc45d6a56. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.848114 26329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:54.850548 26149 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.850920 26149 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:54.852522 26329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.852763 26329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:54.852854 26091 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Refusing update from remote peer d0fb7276976e40d799f5fefcc45d6a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:58:54.853714 26152 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:58:54.876180 26347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.876647 26347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:58:54.877019 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0fb7276976e40d799f5fefcc45d6a56. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:54.877312 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:58:55.222203 26338 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:55.222602 26338 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:55.222793 26338 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:58:55.222883 26338 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:55.232059 26338 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:55.232364 26338 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:55.243829 26338 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39811
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:55.245662 26338 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:55.247247 26338 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:55.258191 26354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:55.258193 26355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:55.258570 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:55.259577 26338 server_base.cc:1061] running on GCE node
I20260812 19:58:55.260337 26338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:55.261693 26338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:55.265100 26338 hybrid_clock.cc:648] HybridClock initialized: now 1786564735264871 us; error 165 us; skew 500 ppm
I20260812 19:58:55.265547 26338 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:55 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:55.269224 26338 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:55.271282 26338 webserver.cc:533] Webserver started at http://127.25.14.1:39811/ using document root <none> and password file <none>
I20260812 19:58:55.272145 26338 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:55.272276 26338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:55.278630 26338 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:58:55.281749 26364 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:55.283270 26338 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:58:55.283489 26338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:55.284250 26338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:55.300490 26338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:55.304680 26338 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:55.305016 26338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:55.306185 26338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:55.308154 26338 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:55.308266 26338 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:55.308436 26338 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:55.308512 26338 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:58:55.308992 26338 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:58:55.309092 26338 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:58:55.309176 26338 fault_injection.cc:59] injecting fault for kudu (pid 26338): FLAGS_fault_before_start (process will exit)
I20260812 19:58:55.339092 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39037
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=39811
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:58:55.367874 26152 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d0fb7276976e40d799f5fefcc45d6a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:58:55.368237 26152 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: LEADER
I20260812 19:58:55.439286 25863 heartbeater.cc:499] Master 127.25.14.60:41211 was elected leader, sending a full tablet report...
W20260812 19:58:55.742592 26375 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:58:55.743036 26375 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:58:55.743244 26375 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:58:55.752149 26375 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:58:55.752427 26375 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:58:55.764147 26375 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36977,127.25.14.21:33835,127.25.14.22:35639,127.25.14.23:37089,127.25.14.24:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=39811
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:58:55.766225 26375 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:58:55.767807 26375 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:58:55.778640 26385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:55.778615 26384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:58:55.779501 26387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:58:55.779902 26375 server_base.cc:1061] running on GCE node
I20260812 19:58:55.780591 26375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:58:55.781927 26375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:58:55.783285 26375 hybrid_clock.cc:648] HybridClock initialized: now 1786564735783123 us; error 118 us; skew 500 ppm
I20260812 19:58:55.783670 26375 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:58:55 dist-test-slave-p3t5 krb5kdc[24842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:58:55.787636 26375 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:58:55.789734 26375 webserver.cc:533] Webserver started at http://127.25.14.1:39811/ using document root <none> and password file <none>
I20260812 19:58:55.790633 26375 fs_manager.cc:362] Metadata directory not provided
I20260812 19:58:55.790828 26375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:58:55.797490 26375 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260812 19:58:55.800828 26395 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:55.802695 26375 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:58:55.802963 26375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:58:55.803761 26375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:58:55.827464 26375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:58:55.831900 26375 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:58:55.832228 26375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:58:55.833340 26375 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:58:55.835117 26375 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:58:55.835250 26375 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:55.835398 26375 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:58:55.835496 26375 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:58:55.878898 26375 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39037
I20260812 19:58:55.878947 26528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39037 every 8 connection(s)
I20260812 19:58:55.880491 26375 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:58:55.889343 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26375
I20260812 19:58:55.889765 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26375
Aug 12 19:58:55 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:58:55 dist-test-slave-p3t5 krb5kdc[24842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
W20260812 19:58:55.914330 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26375: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:39037: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:55.914520 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26375
I20260812 19:58:55.940173 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25708
W20260812 19:58:55.941144 26535 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:55.884266 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:39065 (local address 127.25.14.61:45463)
0812 19:58:55.893089 (+ 8823us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:55.893099 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:55.898781 (+ 5682us) server_negotiation.cc:408] Connection header received
0812 19:58:55.898955 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:55.898961 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:55.899061 (+ 100us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:55.899195 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:55.901567 (+ 2372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:55.903588 (+ 2021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:55.905683 (+ 2095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:55.906205 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:58:55.940051 (+ 33846us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:39065: BlockingRecv error: recv got EOF from 127.25.14.1:39065 (error 108)
Metrics: {"server-negotiator.queue_time_us":8647,"thread_start_us":8505,"threads_started":1}
W20260812 19:58:55.941190 26544 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:55.899632 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:42727 (local address 127.25.14.60:41211)
0812 19:58:55.910500 (+ 10868us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:55.910506 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:55.910537 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:58:55.910603 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:55.910612 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:55.910705 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:55.910831 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:55.912509 (+ 1678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:55.914046 (+ 1537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:55.940014 (+ 25968us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:42727: BlockingRecv error: recv got EOF from 127.25.14.1:42727 (error 108)
Metrics: {"server-negotiator.queue_time_us":10675,"thread_start_us":10576,"threads_started":1}
W20260812 19:58:55.942106 26534 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:55.884297 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:44001 (local address 127.25.14.62:46127)
0812 19:58:55.896562 (+ 12265us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:55.896573 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:55.896604 (+ 31us) server_negotiation.cc:408] Connection header received
0812 19:58:55.896957 (+ 353us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:55.896964 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:55.897070 (+ 106us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:55.897228 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:55.905688 (+ 8460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:55.906968 (+ 1280us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:58:55.909121 (+ 2153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:58:55.909659 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:58:55.913346 (+ 3687us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:58:55.913404 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:58:55.913410 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 19:58:55.913458 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:58:55.921312 (+ 7854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 19:58:55.940196 (+ 18884us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:44001: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":12102,"thread_start_us":11971,"threads_started":1}
W20260812 19:58:55.944550 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25708: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:46491: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:55.944813 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25708
I20260812 19:58:55.970568 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25879
W20260812 19:58:55.973536 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25879: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:32999: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:58:55.973742 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25879
I20260812 19:58:55.996624 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26039
W20260812 19:58:55.999284 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26039: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:46127: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:58:55.999471 26534 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:58:55.997944 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.25.14.62:46127)
0812 19:58:55.998177 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0812 19:58:55.998182 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 19:58:55.998198 (+ 16us) server_negotiation.cc:408] Connection header received
0812 19:58:55.998261 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:58:55.998266 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:58:55.998346 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:58:55.998471 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:58:55.999237 (+ 766us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40216: BlockingRecv error: recv got EOF from 127.0.0.1:40216 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 19:58:55.999472 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26039
I20260812 19:58:56.030546 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25278
I20260812 19:58:56.032052 25346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:56.149000 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25278
I20260812 19:58:56.177845 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 25367
I20260812 19:58:56.179193 25439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:58:56.294202 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 25367
Shutting down hive metastore.
2026-08-12T19:58:56Z chronyd exiting
2026-08-12T19:58:56Z chronyd exiting
2026-08-12T19:58:56Z chronyd exiting
2026-08-12T19:58:56Z chronyd exiting
2026-08-12T19:58:56Z chronyd exiting
Aug 12 19:58:56 dist-test-slave-p3t5 krb5kdc[26557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 19:58:56 dist-test-slave-p3t5 krb5kdc[26557](info): set up 2 sockets
Aug 12 19:58:56 dist-test-slave-p3t5 krb5kdc[26557](info): commencing operation
krb5kdc: starting...
W20260812 19:58:58.884527 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.116s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 19:58:59 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T19:58:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:59Z Disabled control of system clock
2026-08-12T19:58:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:59Z Disabled control of system clock
2026-08-12T19:58:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:59Z Disabled control of system clock
2026-08-12T19:58:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:59Z Disabled control of system clock
2026-08-12T19:58:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:58:59Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 19:59:00: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 19:59:02 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564742, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:59:07 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564747, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 19:59:07 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564747, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.62@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:09.324935 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:59:09.698103 26792 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:09.698520 26792 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:09.698689 26792 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:09.707252 26792 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:09.707371 26792 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:09.707471 26792 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:09.707547 26792 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:09.718896 26792 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:09.720799 26792 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:09.722312 26792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:09.731570 26797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:09.731523 26798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:09.733047 26801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:09.733489 26792 server_base.cc:1061] running on GCE node
I20260812 19:59:09.734382 26792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:09.736122 26792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:09.737377 26792 hybrid_clock.cc:648] HybridClock initialized: now 1786564749737193 us; error 149 us; skew 500 ppm
I20260812 19:59:09.737823 26792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:09 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:09.741396 26792 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:59:09.743386 26792 webserver.cc:533] Webserver started at http://127.25.14.62:35293/ using document root <none> and password file <none>
I20260812 19:59:09.744199 26792 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:09.744345 26792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:09.750303 26792 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:09.753463 26810 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:09.755424 26792 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:09.755656 26792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:59:09.756470 26792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:09.771984 26792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:09.782265 26792 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:09.782610 26792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:09.805545 26792 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:46127
I20260812 19:59:09.805567 26866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:46127 every 8 connection(s)
I20260812 19:59:09.807454 26792 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:09.814283 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26792
I20260812 19:59:09.814558 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:59:09.816478 26867 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:09.822325 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap starting.
I20260812 19:59:09.843128 26867 log.cc:826] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:59:09.855180 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:09.855896 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap complete.
I20260812 19:59:09.867125 26867 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:09.867971 26867 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Initialized, Role: FOLLOWER
I20260812 19:59:09.868986 26867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:09.870085 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:09.870461 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:09.871297 26867 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:09.883312 26886 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:09.883572 26886 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: loading cluster ID for follower catalog manager: success
I20260812 19:59:09.889021 26886 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: acquiring CA information for follower catalog manager: success
I20260812 19:59:09.892014 26886 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Aug 12 19:59:09 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:09.916535 26792 master_runner.cc:187] Error getting master registration for 127.25.14.61:45463: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45463: connect: Connection refused (error 111),
I20260812 19:59:09.920570 26792 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.61@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:09.989343 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45463
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:59:10.404532 26892 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:10.404969 26892 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:10.405129 26892 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:10.414219 26892 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:10.414337 26892 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:10.414460 26892 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:10.414537 26892 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:10.426308 26892 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:10.428149 26892 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:10.429750 26892 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:10.438973 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:10.439074 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:10.439078 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:10.440775 26892 server_base.cc:1061] running on GCE node
I20260812 19:59:10.441866 26892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:10.443809 26892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:10.445199 26892 hybrid_clock.cc:648] HybridClock initialized: now 1786564750445023 us; error 141 us; skew 500 ppm
I20260812 19:59:10.445670 26892 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:10 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:10.449463 26892 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 19:59:10.451402 26892 webserver.cc:533] Webserver started at http://127.25.14.61:43139/ using document root <none> and password file <none>
I20260812 19:59:10.452244 26892 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:10.452407 26892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:10.458861 26892 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:10.462433 26908 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:10.464140 26892 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:10.464401 26892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "99459beb0fb9499e83b30f3c0e1f4854"
format_stamp: "Formatted at 2026-08-12 19:58:49 on dist-test-slave-p3t5"
I20260812 19:59:10.465250 26892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:10.505997 26892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:10.510584 26892 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:10.510926 26892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:10.534288 26892 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:45463
I20260812 19:59:10.534314 26968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:45463 every 8 connection(s)
I20260812 19:59:10.536165 26892 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:59:10.544101 26969 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:10.545941 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 26892
I20260812 19:59:10.546185 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:59:10.548270 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Bootstrap starting.
I20260812 19:59:10.572552 26969 log.cc:826] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:10.586326 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:10.587118 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Bootstrap complete.
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260812 19:59:10.595537 26969 raft_consensus.cc:359] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:10.596379 26969 raft_consensus.cc:740] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99459beb0fb9499e83b30f3c0e1f4854, State: Initialized, Role: FOLLOWER
I20260812 19:59:10.597319 26969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:10.598419 26973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:10.598826 26973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:10.599545 26969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:10.617347 26988 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:10.621497 26988 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: loading cluster ID for follower catalog manager: success
I20260812 19:59:10.626669 26988 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: acquiring CA information for follower catalog manager: success
I20260812 19:59:10.629935 26988 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:59:10 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:10.676076 26892 master_runner.cc:187] Error getting master registration for 127.25.14.60:41211: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41211: connect: Connection refused (error 111),
I20260812 19:59:10.683300 26892 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.60@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:10.729992 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41211
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 19:59:11.164000 27000 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:11.164371 27000 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:11.164466 27000 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:11.173424 27000 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:11.173538 27000 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:11.173594 27000 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:11.173658 27000 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:11.185060 27000 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:11.186890 27000 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:11.188449 27000 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:11.198110 27007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:11.198053 27008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:11.198526 27010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:11.199466 27000 server_base.cc:1061] running on GCE node
I20260812 19:59:11.200493 27000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:11.202452 27000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:11.203877 27000 hybrid_clock.cc:648] HybridClock initialized: now 1786564751203703 us; error 134 us; skew 500 ppm
I20260812 19:59:11.204268 27000 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:11 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564751, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:11.208415 27000 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 19:59:11.211163 27000 webserver.cc:533] Webserver started at http://127.25.14.60:41969/ using document root <none> and password file <none>
I20260812 19:59:11.211982 27000 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:11.212105 27000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:11.218663 27000 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:11.222116 27018 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:11.223889 27000 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:11.224063 27000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "d0fb7276976e40d799f5fefcc45d6a56"
format_stamp: "Formatted at 2026-08-12 19:58:50 on dist-test-slave-p3t5"
I20260812 19:59:11.224809 27000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:11.240387 27000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:11.244397 27000 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:11.244652 27000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:11.266652 27000 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:41211
I20260812 19:59:11.266672 27079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:41211 every 8 connection(s)
I20260812 19:59:11.268553 27000 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:59:11.269177 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27000
I20260812 19:59:11.269464 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:59:11.280351 27080 sys_catalog.cc:263] Verifying existing consensus state
Aug 12 19:59:11 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:11.286216 27080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Bootstrap starting.
Aug 12 19:59:11 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:59:11.305423 27080 log.cc:826] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:11.315903 27080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:11.316557 27080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: Bootstrap complete.
Aug 12 19:59:11 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 19:59:11.323922 27080 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:11.324712 27080 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb7276976e40d799f5fefcc45d6a56, State: Initialized, Role: FOLLOWER
I20260812 19:59:11.325485 27080 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:11.326475 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:11.326958 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:11.327907 27080 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:11.340442 27103 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:11.340579 27103 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: loading cluster ID for follower catalog manager: success
I20260812 19:59:11.345906 27103 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: acquiring CA information for follower catalog manager: success
Aug 12 19:59:11 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564751, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:11.348917 27103 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:11.381176 27000 master_runner.cc:264] No leader master found from master d0fb7276976e40d799f5fefcc45d6a56
I20260812 19:59:11.381318 27000 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:11.559262 27106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:59:11.559572 27106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:11.561820 27106 leader_election.cc:290] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:59:11.569828 26939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854" is_pre_election: true
I20260812 19:59:11.570534 26939 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 2.
I20260812 19:59:11.571332 26813 leader_election.cc:304] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters:
I20260812 19:59:11.571375 27048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56" is_pre_election: true
I20260812 19:59:11.571789 27106 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:59:11.571995 27106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:59:11.572101 27106 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:59:11.572139 27048 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 2.
I20260812 19:59:11.576442 27106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:11.577188 27106 leader_election.cc:290] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 3 election: Requested vote from peers 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:59:11.577790 26939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854"
I20260812 19:59:11.577986 27048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b71e3aac390423aa4f17e84439ded94" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56"
I20260812 19:59:11.578042 26939 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:59:11.578220 27048 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:59:11.583592 26939 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 3.
I20260812 19:59:11.583797 27048 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b71e3aac390423aa4f17e84439ded94 in term 3.
I20260812 19:59:11.584197 26813 leader_election.cc:304] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters:
I20260812 19:59:11.584658 27106 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:59:11.586503 27106 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 LEADER]: Becoming Leader. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Running, Role: LEADER
I20260812 19:59:11.587154 27106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:11.590389 27109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:11.590749 27109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:11.591531 27111 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:11.593573 27111 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:11.594434 27111 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:11.594565 27111 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:11.595777 27111 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:11.596670 27111 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Loaded TSK: 0
I20260812 19:59:11.597666 27111 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:11.769117 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:12.129489 27048 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 3 FOLLOWER]: Refusing update from remote peer 4b71e3aac390423aa4f17e84439ded94: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:59:12.130789 27118 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:12.134569 27119 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.134891 27119 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:12.137212 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.137578 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:12.138857 26939 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Refusing update from remote peer 4b71e3aac390423aa4f17e84439ded94: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:59:12.140389 27119 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.140616 27119 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:12.144341 27118 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:12.149133 27130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b71e3aac390423aa4f17e84439ded94. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.149400 27130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:12.149731 27133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.149966 27133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:12.161087 27116 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:12.161494 27116 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:12.161708 27116 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:12.170524 27116 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:12.170804 27116 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:12.182415 27116 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:12.184257 27116 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:12.186138 27116 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:12.196663 27139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:12.196707 27140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:12.197947 27144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:12.198123 27116 server_base.cc:1061] running on GCE node
I20260812 19:59:12.198849 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:12.200053 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:12.201395 27116 hybrid_clock.cc:648] HybridClock initialized: now 1786564752201215 us; error 138 us; skew 500 ppm
I20260812 19:59:12.201898 27116 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:12 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:12.206187 27116 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:59:12.208041 27116 webserver.cc:533] Webserver started at http://127.25.14.1:45601/ using document root <none> and password file <none>
I20260812 19:59:12.208916 27116 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:12.209062 27116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:12.215741 27116 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:59:12.218957 27152 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.220621 27116 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:59:12.220856 27116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:59:12.221832 27116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:12.239403 27116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:12.244006 27116 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:12.244303 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:12.245402 27116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:12.247215 27116 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:12.247349 27116 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.247490 27116 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:12.247602 27116 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.287949 27116 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34373
I20260812 19:59:12.288002 27275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34373 every 8 connection(s)
I20260812 19:59:12.290292 27116 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:12.299515 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27116
I20260812 19:59:12.299760 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 19:59:12 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:59:12 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:59:12 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:59:12.331028 27277 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:59:12.331473 27278 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:59:12.331485 27277 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:12.331670 27278 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:12.332840 27278 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:59:12.332935 27277 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:59:12.335650 26828 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
I20260812 19:59:12.335741 27036 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
I20260812 19:59:12.338260 26828 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:39499
I20260812 19:59:12.342347 27279 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:59:12.342514 27279 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:12.342857 27279 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:59:12.344708 26928 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
WARNING: no policy specified for kudu/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.2@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.2@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.2 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:12.467558 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:12.570959 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4b71e3aac390423aa4f17e84439ded94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:12.571333 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: LEADER
W20260812 19:59:12.858035 27294 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:12.858461 27294 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:12.858626 27294 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:12.867661 27294 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:12.867951 27294 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 19:59:12.879763 27294 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:12.881750 27294 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:12.883577 27294 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:12.895352 27300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:12.895397 27299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:12.895483 27302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:12.896817 27294 server_base.cc:1061] running on GCE node
I20260812 19:59:12.897565 27294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:12.898865 27294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:12.902204 27294 hybrid_clock.cc:648] HybridClock initialized: now 1786564752902011 us; error 146 us; skew 500 ppm
I20260812 19:59:12.902608 27294 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:12 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:12.907198 27294 init.cc:377] Logged in from keytab as kudu/127.25.14.2@KRBTEST.COM (short username kudu)
I20260812 19:59:12.909154 27294 webserver.cc:533] Webserver started at http://127.25.14.2:42219/ using document root <none> and password file <none>
I20260812 19:59:12.910127 27294 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:12.910300 27294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:12.916493 27294 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:59:12.919772 27310 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.921233 27294 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:59:12.921450 27294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b"
format_stamp: "Formatted at 2026-08-12 19:58:52 on dist-test-slave-p3t5"
I20260812 19:59:12.922366 27294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:12.938490 27294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:12.943120 27294 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:12.943470 27294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:12.944613 27294 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:12.946574 27294 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:12.946734 27294 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.946869 27294 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:12.946962 27294 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:12.983891 27294 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:36161
I20260812 19:59:12.983973 27442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:36161 every 8 connection(s)
I20260812 19:59:12.985724 27294 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 19:59:12.987068 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27294
I20260812 19:59:12.987347 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.2@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:59:13.025027 27445 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:59:13.025497 27445 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.026114 27443 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:59:13.026252 27443 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.026868 27445 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:59:13.027118 27443 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:59:13.029131 26828 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:36161)
I20260812 19:59:13.029207 27036 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:36161)
I20260812 19:59:13.030357 26828 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:44383
I20260812 19:59:13.032618 27447 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:59:13.032819 27447 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.033324 27447 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:59:13.034516 26928 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:36161)
WARNING: no policy specified for kudu/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.3@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.14.3@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 19:59:13.160849 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:13.341408 27278 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
W20260812 19:59:13.555470 27462 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:13.555886 27462 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:13.556054 27462 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:13.565898 27462 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:13.566226 27462 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 19:59:13.578682 27462 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:13.580443 27462 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:13.582062 27462 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:13.592536 27467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:13.592567 27468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:13.594295 27470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:13.594295 27462 server_base.cc:1061] running on GCE node
I20260812 19:59:13.595098 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:13.596302 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:13.597661 27462 hybrid_clock.cc:648] HybridClock initialized: now 1786564753597512 us; error 81 us; skew 500 ppm
I20260812 19:59:13.598052 27462 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:13.602087 27462 init.cc:377] Logged in from keytab as kudu/127.25.14.3@KRBTEST.COM (short username kudu)
I20260812 19:59:13.604058 27462 webserver.cc:533] Webserver started at http://127.25.14.3:33833/ using document root <none> and password file <none>
I20260812 19:59:13.604882 27462 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:13.605047 27462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:13.611163 27462 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 19:59:13.614338 27477 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:13.616485 27462 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:13.616693 27462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "baa124c27c1249b59575d27413852084"
format_stamp: "Formatted at 2026-08-12 19:58:53 on dist-test-slave-p3t5"
I20260812 19:59:13.617514 27462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:13.639937 27462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:13.644337 27462 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:13.644649 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:13.645846 27462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:13.647725 27462 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:13.647882 27462 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:13.648028 27462 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:13.648123 27462 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:13.686910 27462 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:36127
I20260812 19:59:13.687017 27607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:36127 every 8 connection(s)
I20260812 19:59:13.688687 27462 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 19:59:13.690594 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27462
I20260812 19:59:13.690824 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 19:59:13 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.3@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:13.738802 27608 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:59:13.739248 27608 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.740314 27608 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:59:13.738785 27610 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:59:13.741509 27610 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.741958 27611 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:59:13.742128 27611 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:13.742216 27610 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:59:13.742218 27036 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:36127)
I20260812 19:59:13.742669 27611 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:59:13.743552 26828 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:36127)
I20260812 19:59:13.743899 26928 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:36127)
I20260812 19:59:13.744591 26828 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:51621
I20260812 19:59:13.754976 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:59:13.763016 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 19:59:13.763253 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26792
I20260812 19:59:13.764199 26861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:13.879199 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26792
W20260812 19:59:13.908012 27021 connection.cc:573] server connection from 127.0.0.1:44518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:13.909602 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--webserver_interface=127.25.14.62
--webserver_port=35293
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:14.033607 27445 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
W20260812 19:59:14.036168 27445 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:46127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:46127: connect: Connection refused (error 111)
W20260812 19:59:14.299383 27621 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:14.299754 27621 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:14.299897 27621 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:14.308786 27621 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:14.308910 27621 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:14.309010 27621 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:14.309096 27621 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:14.320747 27621 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35293
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:14.322719 27621 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:14.324357 27621 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:14.333822 27627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:14.334025 27628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:14.334268 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:14.335841 27621 server_base.cc:1061] running on GCE node
I20260812 19:59:14.336763 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:14.338680 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:14.340014 27621 hybrid_clock.cc:648] HybridClock initialized: now 1786564754339848 us; error 129 us; skew 500 ppm
I20260812 19:59:14.340409 27621 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260812 19:59:14.345266 27278 heartbeater.cc:646] Failed to heartbeat to 127.25.14.62:46127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.14.62:46127: connect: Connection refused (error 111)
Aug 12 19:59:14 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:14.353987 27621 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 19:59:14.355912 27621 webserver.cc:533] Webserver started at http://127.25.14.62:35293/ using document root <none> and password file <none>
I20260812 19:59:14.356753 27621 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:14.356869 27621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:14.362933 27621 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:14.366398 27640 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:14.368309 27621 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 19:59:14.368532 27621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "4b71e3aac390423aa4f17e84439ded94"
format_stamp: "Formatted at 2026-08-12 19:58:48 on dist-test-slave-p3t5"
I20260812 19:59:14.369401 27621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:14.392920 27621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:14.397032 27621 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:14.397346 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:14.420626 27621 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:46127
I20260812 19:59:14.420637 27696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:46127 every 8 connection(s)
I20260812 19:59:14.422631 27621 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:14.428973 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27621
I20260812 19:59:14.429395 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27116
I20260812 19:59:14.431185 27697 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:14.436161 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap starting.
Aug 12 19:59:14 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 19:59:14.451485 27271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:14.458217 27697 log.cc:826] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:14.468319 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:14.468935 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: Bootstrap complete.
I20260812 19:59:14.476666 27697 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:14.477458 27697 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b71e3aac390423aa4f17e84439ded94, State: Initialized, Role: FOLLOWER
I20260812 19:59:14.478400 27697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:14.479446 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:14.479775 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:14.480798 27697 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:14.492648 27717 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:14.492787 27717 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: loading cluster ID for follower catalog manager: success
I20260812 19:59:14.498359 27717 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: acquiring CA information for follower catalog manager: success
I20260812 19:59:14.501636 27717 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 19:59:14 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:14.571679 27621 master_runner.cc:264] No leader master found from master 4b71e3aac390423aa4f17e84439ded94
I20260812 19:59:14.571882 27621 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:14.572732 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27116
I20260812 19:59:14.608788 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34373
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=45601
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:14.747174 27610 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
I20260812 19:59:14.758865 27658 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "baa124c27c1249b59575d27413852084" instance_seqno: 1786564753677309) as {username='kudu', principal='kudu/127.25.14.3@KRBTEST.COM'} at 127.25.14.3:44865; Asking this server to re-register.
I20260812 19:59:14.760141 27610 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:14.762195 27658 ts_manager.cc:194] Registered new tserver with Master: baa124c27c1249b59575d27413852084 (127.25.14.3:36127)
W20260812 19:59:15.038399 27725 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:15.038949 27725 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:15.039136 27725 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:59:15.043838 27445 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:59:15.044085 27445 heartbeater.cc:499] Master 127.25.14.62:46127 was elected leader, sending a full tablet report...
I20260812 19:59:15.045187 27658 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a7a6aaed8c694d6c9c47a4fdde6eb81b" instance_seqno: 1786564752974871) as {username='kudu', principal='kudu/127.25.14.2@KRBTEST.COM'} at 127.25.14.2:32817; Asking this server to re-register.
I20260812 19:59:15.045888 27445 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:15.047114 27658 ts_manager.cc:194] Registered new tserver with Master: a7a6aaed8c694d6c9c47a4fdde6eb81b (127.25.14.2:36161)
W20260812 19:59:15.049461 27725 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:15.049708 27725 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:15.061112 27725 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45601
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:15.062943 27725 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:15.064368 27725 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:15.075239 27732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:15.075877 27735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:15.076072 27733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:15.076797 27725 server_base.cc:1061] running on GCE node
I20260812 19:59:15.077474 27725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:15.078667 27725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:15.079923 27725 hybrid_clock.cc:648] HybridClock initialized: now 1786564755079771 us; error 119 us; skew 500 ppm
I20260812 19:59:15.080282 27725 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:15.084547 27725 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:59:15.086686 27725 webserver.cc:533] Webserver started at http://127.25.14.1:45601/ using document root <none> and password file <none>
I20260812 19:59:15.087589 27725 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:15.087699 27725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:15.094226 27725 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:15.097431 27742 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:15.099217 27725 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:15.099424 27725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:59:15.100297 27725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:15.125874 27725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:15.134590 27725 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:15.134922 27725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:15.136049 27725 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:15.137964 27725 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:15.138087 27725 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:15.138221 27725 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:15.138317 27725 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:15.174575 27725 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34373
I20260812 19:59:15.174595 27871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34373 every 8 connection(s)
I20260812 19:59:15.176352 27725 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:15.179291 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27725
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
I20260812 19:59:15.198186 27881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:59:15.198643 27881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 19:59:15.219197 27881 leader_election.cc:290] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 19:59:15.234737 27048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56" is_pre_election: true
I20260812 19:59:15.239570 27668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94" is_pre_election: true
I20260812 19:59:15.240317 27668 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99459beb0fb9499e83b30f3c0e1f4854 in term 3.
I20260812 19:59:15.241871 26912 leader_election.cc:304] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters: d0fb7276976e40d799f5fefcc45d6a56
I20260812 19:59:15.242128 27874 heartbeater.cc:344] Connected to a master server at 127.25.14.62:46127
I20260812 19:59:15.242522 27874 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:15.242735 27881 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:59:15.242867 27881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:59:15.242990 27881 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:59:15.243695 27874 heartbeater.cc:507] Master 127.25.14.62:46127 requested a full tablet report, sending...
I20260812 19:59:15.245931 27658 ts_manager.cc:194] Registered new tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
I20260812 19:59:15.246155 27881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:15.247219 27875 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45463
I20260812 19:59:15.247401 27875 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:15.248028 27875 heartbeater.cc:507] Master 127.25.14.61:45463 requested a full tablet report, sending...
I20260812 19:59:15.248137 27668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94"
I20260812 19:59:15.248380 27668 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:59:15.248425 27048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99459beb0fb9499e83b30f3c0e1f4854" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0fb7276976e40d799f5fefcc45d6a56"
I20260812 19:59:15.250543 26928 ts_manager.cc:194] Re-registered known tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
I20260812 19:59:15.250993 27881 leader_election.cc:290] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 4 election: Requested vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), d0fb7276976e40d799f5fefcc45d6a56 (127.25.14.60:41211)
I20260812 19:59:15.252990 27872 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41211
I20260812 19:59:15.253192 27872 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:15.253783 27668 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99459beb0fb9499e83b30f3c0e1f4854 in term 4.
I20260812 19:59:15.253826 27872 heartbeater.cc:507] Master 127.25.14.60:41211 requested a full tablet report, sending...
I20260812 19:59:15.254280 26912 leader_election.cc:304] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854; no voters: d0fb7276976e40d799f5fefcc45d6a56
I20260812 19:59:15.254690 27881 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:59:15.255298 27036 ts_manager.cc:194] Re-registered known tserver with Master: 88901ce63f4245549c8d698e4c96b911 (127.25.14.1:34373)
I20260812 19:59:15.256114 27881 raft_consensus.cc:697] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [term 4 LEADER]: Becoming Leader. State: Replica: 99459beb0fb9499e83b30f3c0e1f4854, State: Running, Role: LEADER
I20260812 19:59:15.256696 27881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:15.259179 27890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99459beb0fb9499e83b30f3c0e1f4854. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.259560 27890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:15.260478 27892 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:15.264429 27892 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:15.265697 27892 catalog_manager.cc:1295] Loaded cluster ID: a49dd629122f45db9df84bbe09ea83f1
I20260812 19:59:15.265823 27892 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:15.267529 27892 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:15.268663 27892 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854: Loaded TSK: 0
I20260812 19:59:15.270068 27892 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:59:15.381083 27895 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4b71e3aac390423aa4f17e84439ded94)
I20260812 19:59:15.381421 27895 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } }
I20260812 19:59:15.383697 27895 leader_election.cc:290] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4b71e3aac390423aa4f17e84439ded94 (127.25.14.62:46127), 99459beb0fb9499e83b30f3c0e1f4854 (127.25.14.61:45463)
I20260812 19:59:15.384558 26939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99459beb0fb9499e83b30f3c0e1f4854" is_pre_election: true
I20260812 19:59:15.388652 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27725
W20260812 19:59:15.390061 27898 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:15.385905 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.25.14.1:34373)
0812 19:59:15.386744 (+ 839us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:15.386751 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:15.386781 (+ 30us) server_negotiation.cc:408] Connection header received
0812 19:59:15.386839 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:15.386846 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:15.387074 (+ 228us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:15.387299 (+ 225us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:15.388397 (+ 1098us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":468,"threads_started":1}
I20260812 19:59:15.391799 27668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0fb7276976e40d799f5fefcc45d6a56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4b71e3aac390423aa4f17e84439ded94" is_pre_election: true
W20260812 19:59:15.392002 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27725: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34373: server requires authentication, but client does not have Kerberos credentials available
W20260812 19:59:15.392134 27898 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:15.389515 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.25.14.1:34373)
0812 19:59:15.390416 (+ 901us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:15.390424 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:15.390725 (+ 301us) server_negotiation.cc:408] Connection header received
0812 19:59:15.390807 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:15.390813 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:15.390898 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:15.391006 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:15.391900 (+ 894us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35374: BlockingRecv error: recv got EOF from 127.0.0.1:35374 (error 108)
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":1132,"threads_started":1}
I20260812 19:59:15.392160 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27725
I20260812 19:59:15.392251 27668 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0fb7276976e40d799f5fefcc45d6a56 in current term 4: Already voted for candidate 99459beb0fb9499e83b30f3c0e1f4854 in this term.
I20260812 19:59:15.396347 27022 leader_election.cc:304] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0fb7276976e40d799f5fefcc45d6a56; no voters: 4b71e3aac390423aa4f17e84439ded94, 99459beb0fb9499e83b30f3c0e1f4854
I20260812 19:59:15.396875 27895 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:59:15.399261 27895 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:59:15.418081 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34373
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=45601
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:15.809713 27048 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [term 4 FOLLOWER]: Refusing update from remote peer 99459beb0fb9499e83b30f3c0e1f4854: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260812 19:59:15.810588 27906 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 19:59:15.812750 27895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99459beb0fb9499e83b30f3c0e1f4854. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.813027 27895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:15.815157 27909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.815523 27909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:15.816756 27668 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [term 4 FOLLOWER]: Refusing update from remote peer 99459beb0fb9499e83b30f3c0e1f4854: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260812 19:59:15.820598 27900 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:15.821334 27900 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:15.821656 27900 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:59:15.821846 27900 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:59:15.822595 27909 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 19:59:15.824816 27895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.825277 27895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fb7276976e40d799f5fefcc45d6a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:15.828224 27918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.828548 27918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:15.829066 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99459beb0fb9499e83b30f3c0e1f4854. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:15.829337 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b71e3aac390423aa4f17e84439ded94 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:15.834584 27900 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:15.834887 27900 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:15.846338 27900 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45601
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:15.848120 27900 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:15.849947 27900 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:15.860961 27921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:15.860895 27920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:15.861199 27923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:15.861716 27900 server_base.cc:1061] running on GCE node
I20260812 19:59:15.862464 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:15.863786 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:15.865150 27900 hybrid_clock.cc:648] HybridClock initialized: now 1786564755864959 us; error 137 us; skew 500 ppm
I20260812 19:59:15.865494 27900 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:15 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:15.869515 27900 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:59:15.871505 27900 webserver.cc:533] Webserver started at http://127.25.14.1:45601/ using document root <none> and password file <none>
I20260812 19:59:15.872375 27900 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:15.872505 27900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:15.878896 27900 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:15.882186 27932 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:15.884007 27900 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 19:59:15.884294 27900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:59:15.885210 27900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:15.908417 27900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:15.912948 27900 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:15.913295 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:15.914407 27900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:15.916141 27900 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:15.916257 27900 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:15.916414 27900 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:15.916489 27900 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:59:15.916918 27900 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:59:15.917014 27900 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:59:15.917101 27900 fault_injection.cc:59] injecting fault for kudu (pid 27900): FLAGS_fault_before_start (process will exit)
I20260812 19:59:15.942044 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34373
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--webserver_port=45601
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 19:59:16.290037 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "99459beb0fb9499e83b30f3c0e1f4854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b71e3aac390423aa4f17e84439ded94" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 46127 } } peers { permanent_uuid: "99459beb0fb9499e83b30f3c0e1f4854" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45463 } } peers { permanent_uuid: "d0fb7276976e40d799f5fefcc45d6a56" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41211 } } }
I20260812 19:59:16.290411 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99459beb0fb9499e83b30f3c0e1f4854 [sys.catalog]: This master's current role is: LEADER
W20260812 19:59:16.340504 27941 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:16.340961 27941 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:16.341066 27941 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:16.349952 27941 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:16.350168 27941 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:16.361934 27941 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44953,127.25.14.21:34551,127.25.14.22:37813,127.25.14.23:33285,127.25.14.24:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45601
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:46127,127.25.14.61:45463,127.25.14.60:41211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:16.363672 27941 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:16.365134 27941 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:16.376390 27948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:16.376363 27947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:16.377132 27950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:16.379225 27941 server_base.cc:1061] running on GCE node
I20260812 19:59:16.379922 27941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:16.381167 27941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:16.384599 27941 hybrid_clock.cc:648] HybridClock initialized: now 1786564756384366 us; error 160 us; skew 500 ppm
I20260812 19:59:16.385131 27941 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 19:59:16 dist-test-slave-p3t5 krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 19:59:16.390518 27941 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 19:59:16.392483 27941 webserver.cc:533] Webserver started at http://127.25.14.1:45601/ using document root <none> and password file <none>
I20260812 19:59:16.393350 27941 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:16.393512 27941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:16.399609 27941 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:59:16.403183 27959 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:16.405017 27941 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:16.405226 27941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "88901ce63f4245549c8d698e4c96b911"
format_stamp: "Formatted at 2026-08-12 19:58:51 on dist-test-slave-p3t5"
I20260812 19:59:16.406138 27941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:16.429270 27941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:16.433902 27941 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:16.434217 27941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:16.435374 27941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:16.437295 27941 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:16.437422 27941 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:16.437562 27941 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:16.437695 27941 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 19:59:16.475174 27941 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34373
I20260812 19:59:16.475291 28081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34373 every 8 connection(s)
I20260812 19:59:16.476938 27941 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:16.484083 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 27941
I20260812 19:59:16.484477 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27941
Aug 12 19:59:16 dist-test-slave-p3t5 krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
W20260812 19:59:16.503108 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27941: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34373: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:59:16.503280 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27941
I20260812 19:59:16.526499 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27294
W20260812 19:59:16.527454 28091 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:16.485124 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:36181 (local address 127.25.14.60:41211)
0812 19:59:16.497383 (+ 12259us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:16.497389 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:16.499065 (+ 1676us) server_negotiation.cc:408] Connection header received
0812 19:59:16.499214 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:16.499226 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:16.499344 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:16.499480 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:16.502373 (+ 2893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:16.505409 (+ 3036us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:16.526258 (+ 20849us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:36181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":12119,"thread_start_us":13097,"threads_started":1}
W20260812 19:59:16.527469 28086 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:16.480650 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:39511 (local address 127.25.14.61:45463)
0812 19:59:16.481972 (+ 1322us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:16.481981 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:16.483037 (+ 1056us) server_negotiation.cc:408] Connection header received
0812 19:59:16.487136 (+ 4099us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:16.487148 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:16.487289 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:16.487462 (+ 173us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:16.492352 (+ 4890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:16.494453 (+ 2101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:16.497068 (+ 2615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:16.497736 (+ 668us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:59:16.526334 (+ 28598us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:39511: BlockingRecv error: recv got EOF from 127.25.14.1:39511 (error 108)
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":1024,"threads_started":1}
W20260812 19:59:16.528071 28087 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:16.480650 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:41159 (local address 127.25.14.62:46127)
0812 19:59:16.482449 (+ 1799us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:16.482456 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:16.482976 (+ 520us) server_negotiation.cc:408] Connection header received
0812 19:59:16.486064 (+ 3088us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:16.486075 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:16.486210 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:16.486356 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:16.492587 (+ 6231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:16.494696 (+ 2109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:16.501054 (+ 6358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:16.501685 (+ 631us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:59:16.527208 (+ 25523us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:41159: BlockingRecv error: recv got EOF from 127.25.14.1:41159 (error 108)
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1522,"threads_started":1}
W20260812 19:59:16.530227 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27294: Not authorized: Client connection negotiation failed: client connection to 127.25.14.2:36161: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:59:16.530395 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27294
I20260812 19:59:16.556119 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27462
W20260812 19:59:16.563165 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27462: Not authorized: Client connection negotiation failed: client connection to 127.25.14.3:36127: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:59:16.563548 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27462
I20260812 19:59:16.592537 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27621
W20260812 19:59:16.595451 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27621: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:46127: server requires authentication, but client does not have Kerberos credentials available
I20260812 19:59:16.595630 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27621
I20260812 19:59:16.626135 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 26892
I20260812 19:59:16.627619 26963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:16.744767 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 26892
I20260812 19:59:16.774379 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 27000
I20260812 19:59:16.775906 27074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:16.888334 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 27000
Shutting down hive metastore.
2026-08-12T19:59:17Z chronyd exiting
2026-08-12T19:59:17Z chronyd exiting
2026-08-12T19:59:17Z chronyd exiting
2026-08-12T19:59:17Z chronyd exiting
2026-08-12T19:59:17Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/11 (48738 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/0
I20260812 19:59:17.383282 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34481
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537 with env {}
W20260812 19:59:17.771992 28107 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:17.772380 28107 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:17.772455 28107 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:17.781213 28107 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:17.781342 28107 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:17.781458 28107 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:17.781533 28107 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:17.792605 28107 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:17.794516 28107 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:17.796509 28107 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:17.806177 28112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:17.806231 28113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:17.807812 28115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:17.807900 28107 server_base.cc:1061] running on GCE node
I20260812 19:59:17.808872 28107 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:17.809110 28107 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:17.809221 28107 hybrid_clock.cc:648] HybridClock initialized: now 1786564757809177 us; error 0 us; skew 500 ppm
I20260812 19:59:17.809674 28107 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:17.813135 28107 webserver.cc:533] Webserver started at http://127.25.14.62:37531/ using document root <none> and password file <none>
I20260812 19:59:17.814138 28107 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:17.814244 28107 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:17.814680 28107 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:17.817253 28107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "46e5719344eb452085fe80ccfecd51e3"
format_stamp: "Formatted at 2026-08-12 19:59:17 on dist-test-slave-p3t5"
I20260812 19:59:17.818197 28107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "46e5719344eb452085fe80ccfecd51e3"
format_stamp: "Formatted at 2026-08-12 19:59:17 on dist-test-slave-p3t5"
I20260812 19:59:17.824271 28107 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:59:17.828315 28121 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:17.830057 28107 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 19:59:17.830343 28107 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "46e5719344eb452085fe80ccfecd51e3"
format_stamp: "Formatted at 2026-08-12 19:59:17 on dist-test-slave-p3t5"
I20260812 19:59:17.830586 28107 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:17.843547 28107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:17.844586 28107 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:17.844906 28107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:17.868639 28107 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34481
I20260812 19:59:17.868690 28176 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34481 every 8 connection(s)
I20260812 19:59:17.870579 28107 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:17.873941 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28107
I20260812 19:59:17.874204 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:59:17.875463 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:34495
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537 with env {}
I20260812 19:59:17.875751 28177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:17.882503 28177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:17.904018 28177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:17.905510 28125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
W20260812 19:59:17.907565 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34495: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
I20260812 19:59:17.957119 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } attempt: 1
W20260812 19:59:17.959587 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34495: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
I20260812 19:59:18.027938 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } attempt: 2
W20260812 19:59:18.030349 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34495: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
I20260812 19:59:18.121717 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } attempt: 3
W20260812 19:59:18.124420 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34495: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
I20260812 19:59:18.267753 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } attempt: 4
W20260812 19:59:18.270453 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34495: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34495: connect: Connection refused (error 111)
W20260812 19:59:18.279912 28181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:18.280318 28181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:18.280474 28181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:18.289435 28181 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:18.289554 28181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:18.289682 28181 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:18.289796 28181 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:18.301165 28181 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:34495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:18.303078 28181 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:18.305182 28181 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:18.314792 28187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:18.315275 28188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:18.315971 28190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:18.316949 28181 server_base.cc:1061] running on GCE node
I20260812 19:59:18.317972 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:18.318183 28181 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:18.318292 28181 hybrid_clock.cc:648] HybridClock initialized: now 1786564758318251 us; error 0 us; skew 500 ppm
I20260812 19:59:18.318681 28181 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:18.321983 28181 webserver.cc:533] Webserver started at http://127.25.14.61:40913/ using document root <none> and password file <none>
I20260812 19:59:18.322847 28181 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:18.322948 28181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:18.323395 28181 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:18.330132 28181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "a21389c33ea540dd8781308e71e64efb"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.331079 28181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "a21389c33ea540dd8781308e71e64efb"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.337513 28181 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260812 19:59:18.341549 28195 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:18.343158 28181 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:18.343403 28181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "a21389c33ea540dd8781308e71e64efb"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.343660 28181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:18.384641 28181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:18.385833 28181 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:18.386170 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:18.409505 28181 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:34495
I20260812 19:59:18.409513 28254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:34495 every 8 connection(s)
I20260812 19:59:18.411378 28181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:59:18.416859 28255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:18.420411 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28181
I20260812 19:59:18.420640 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:59:18.421831 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:34537
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537 with env {}
I20260812 19:59:18.424355 28255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:18.440321 28255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:18.452356 28255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:18.453907 28196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
W20260812 19:59:18.455966 28255 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.505400 28255 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 1
W20260812 19:59:18.508038 28255 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.569772 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } attempt: 5
I20260812 19:59:18.576397 28255 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 2
I20260812 19:59:18.577724 28177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:18.579192 28255 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
W20260812 19:59:18.579962 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.631311 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 1
W20260812 19:59:18.634155 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.670451 28255 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 3
W20260812 19:59:18.673184 28255 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.702487 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 2
W20260812 19:59:18.705118 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.811408 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 3
W20260812 19:59:18.814317 28177 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
I20260812 19:59:18.816452 28255 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 4
W20260812 19:59:18.818704 28255 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34537: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34537: connect: Connection refused (error 111)
W20260812 19:59:18.829085 28259 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:18.829548 28259 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:18.829720 28259 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:18.838721 28259 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:18.838874 28259 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:18.839008 28259 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:18.839100 28259 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:18.850149 28259 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:18.852089 28259 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:18.854202 28259 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:18.863487 28268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:18.863517 28267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:18.864540 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:18.864897 28259 server_base.cc:1061] running on GCE node
I20260812 19:59:18.865912 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:18.866119 28259 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:18.866243 28259 hybrid_clock.cc:648] HybridClock initialized: now 1786564758866199 us; error 0 us; skew 500 ppm
I20260812 19:59:18.866642 28259 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:18.870036 28259 webserver.cc:533] Webserver started at http://127.25.14.60:35469/ using document root <none> and password file <none>
I20260812 19:59:18.870893 28259 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:18.871049 28259 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:18.871441 28259 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:18.874027 28259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "6404a0d960084969a07548b0477d58c6"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.874890 28259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "6404a0d960084969a07548b0477d58c6"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.880877 28259 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:59:18.885147 28276 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:18.886829 28259 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:18.887053 28259 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "6404a0d960084969a07548b0477d58c6"
format_stamp: "Formatted at 2026-08-12 19:59:18 on dist-test-slave-p3t5"
I20260812 19:59:18.887311 28259 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:18.906739 28259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:18.907795 28259 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:18.908120 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:18.931118 28259 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:34537
I20260812 19:59:18.931140 28336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:34537 every 8 connection(s)
I20260812 19:59:18.932902 28259 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:59:18.934959 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28259
I20260812 19:59:18.935186 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:59:18.938977 28337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:18.948001 28337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:18.962598 28337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:18.969488 28337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:18.981187 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: Bootstrap starting.
I20260812 19:59:18.984764 28337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:18.986102 28337 log.cc:826] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:18.989358 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: No bootstrap required, opened a new log
I20260812 19:59:18.991590 28177 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 4
I20260812 19:59:18.996212 28337 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:18.996668 28337 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:18.996892 28337 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6404a0d960084969a07548b0477d58c6, State: Initialized, Role: FOLLOWER
I20260812 19:59:18.997833 28337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:18.999174 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:18.999495 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.000578 28337 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:19.004988 28177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Bootstrap starting.
I20260812 19:59:19.011171 28177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Neither blocks nor log segments found. Creating new log.
W20260812 19:59:19.011886 28354 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:19.012019 28354 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:19.012467 28177 log.cc:826] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:19.015259 28177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: No bootstrap required, opened a new log
I20260812 19:59:19.009289 28259 master_runner.cc:187] Error getting master registration for 127.25.14.62:34481: OK, instance_id { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" instance_seqno: 1786564757862366 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:19.016248 28259 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:19.020843 28177 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.021173 28177 raft_consensus.cc:385] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:19.021312 28177 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e5719344eb452085fe80ccfecd51e3, State: Initialized, Role: FOLLOWER
I20260812 19:59:19.022151 28177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.024263 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.024616 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.025490 28177 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:19.037043 28371 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:19.037292 28371 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:19.035437 28107 master_runner.cc:187] Error getting master registration for 127.25.14.61:34495: OK, instance_id { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" instance_seqno: 1786564758403205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:19.042142 28107 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:59:19.098436 28127 tablet.cc:2406] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:19.117935 28255 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } attempt: 5
I20260812 19:59:19.131832 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: Bootstrap starting.
I20260812 19:59:19.137283 28255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:19.139052 28255 log.cc:826] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:19.142670 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: No bootstrap required, opened a new log
I20260812 19:59:19.151338 28255 raft_consensus.cc:359] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.151762 28255 raft_consensus.cc:385] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:19.151923 28255 raft_consensus.cc:740] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21389c33ea540dd8781308e71e64efb, State: Initialized, Role: FOLLOWER
I20260812 19:59:19.152760 28255 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.154033 28375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.154466 28375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.155479 28255 sys_catalog.cc:565] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:19.160604 28283 tablet.cc:2406] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:19.208314 28181 master_runner.cc:264] No leader master found from master a21389c33ea540dd8781308e71e64efb
I20260812 19:59:19.208482 28181 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:59:19.212838 28386 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:19.213029 28386 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:19.223508 28375 raft_consensus.cc:493] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:59:19.223850 28375 raft_consensus.cc:515] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.226003 28375 leader_election.cc:290] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46e5719344eb452085fe80ccfecd51e3 (127.25.14.62:34481), 6404a0d960084969a07548b0477d58c6 (127.25.14.60:34537)
I20260812 19:59:19.231206 28310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21389c33ea540dd8781308e71e64efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6404a0d960084969a07548b0477d58c6" is_pre_election: true
I20260812 19:59:19.231868 28310 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21389c33ea540dd8781308e71e64efb in term 0.
I20260812 19:59:19.232877 28196 leader_election.cc:304] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6404a0d960084969a07548b0477d58c6, a21389c33ea540dd8781308e71e64efb; no voters:
I20260812 19:59:19.233999 28150 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21389c33ea540dd8781308e71e64efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e5719344eb452085fe80ccfecd51e3" is_pre_election: true
I20260812 19:59:19.234587 28150 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21389c33ea540dd8781308e71e64efb in term 0.
I20260812 19:59:19.236337 28375 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:59:19.236554 28375 raft_consensus.cc:493] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:59:19.236670 28375 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:19.240315 28375 raft_consensus.cc:515] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.241928 28150 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21389c33ea540dd8781308e71e64efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e5719344eb452085fe80ccfecd51e3"
I20260812 19:59:19.242187 28150 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:19.246650 28150 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21389c33ea540dd8781308e71e64efb in term 1.
I20260812 19:59:19.247478 28200 leader_election.cc:304] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e5719344eb452085fe80ccfecd51e3, a21389c33ea540dd8781308e71e64efb; no voters:
I20260812 19:59:19.250901 28310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21389c33ea540dd8781308e71e64efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6404a0d960084969a07548b0477d58c6"
I20260812 19:59:19.251187 28310 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:19.255239 28310 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21389c33ea540dd8781308e71e64efb in term 1.
I20260812 19:59:19.259970 28375 leader_election.cc:290] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [CANDIDATE]: Term 1 election: Requested vote from peers 46e5719344eb452085fe80ccfecd51e3 (127.25.14.62:34481), 6404a0d960084969a07548b0477d58c6 (127.25.14.60:34537)
I20260812 19:59:19.260614 28375 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:59:19.261236 28375 raft_consensus.cc:697] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 1 LEADER]: Becoming Leader. State: Replica: a21389c33ea540dd8781308e71e64efb, State: Running, Role: LEADER
I20260812 19:59:19.263087 28375 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } }
I20260812 19:59:19.277137 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21389c33ea540dd8781308e71e64efb. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.277657 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:19.284909 28391 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:19.287866 28391 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:19.310134 28310 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:19.311522 28388 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:19.320322 28150 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:19.321635 28388 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:19.365052 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21389c33ea540dd8781308e71e64efb. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.365391 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.371869 28392 mvcc.cc:204] Tried to move back new op lower bound from 7317769254119669760 to 7317769253957861376. Current Snapshot: MvccSnapshot[applied={T|T < 7317769254119669760}]
I20260812 19:59:19.385460 28394 mvcc.cc:204] Tried to move back new op lower bound from 7317769254119669760 to 7317769253957861376. Current Snapshot: MvccSnapshot[applied={T|T < 7317769254119669760}]
I20260812 19:59:19.394644 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21389c33ea540dd8781308e71e64efb. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.394996 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.403669 28393 mvcc.cc:204] Tried to move back new op lower bound from 7317769254119669760 to 7317769253957861376. Current Snapshot: MvccSnapshot[applied={T|T < 7317769254119669760}]
I20260812 19:59:19.404492 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.404773 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.408006 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.408273 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:19.409503 28391 catalog_manager.cc:1383] Generated new cluster ID: 2bd2b77dccea4273857b1184baf6b455
I20260812 19:59:19.409675 28391 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:19.431939 28375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.432265 28375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:19.432741 28375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } }
I20260812 19:59:19.432969 28375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:19.452900 28391 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:59:19.454861 28391 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:19.486057 28391 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: Generated new TSK 0
I20260812 19:59:19.487469 28391 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:59:19.499465 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--time_source=system_unsync with env {}
W20260812 19:59:19.970991 28403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:19.971412 28403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:19.971542 28403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:19.980418 28403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:19.980659 28403 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:19.992030 28403 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:19.994027 28403 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:19.996107 28403 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:20.008033 28409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:20.007997 28412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:20.008056 28410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:20.009203 28403 server_base.cc:1061] running on GCE node
I20260812 19:59:20.010033 28403 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:20.010267 28403 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:20.010375 28403 hybrid_clock.cc:648] HybridClock initialized: now 1786564760010351 us; error 0 us; skew 500 ppm
I20260812 19:59:20.010735 28403 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:20.013419 28403 webserver.cc:533] Webserver started at http://127.25.14.1:44297/ using document root <none> and password file <none>
I20260812 19:59:20.014619 28354 catalog_manager.cc:1295] Loaded cluster ID: 2bd2b77dccea4273857b1184baf6b455
I20260812 19:59:20.014621 28403 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:20.014722 28354 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: loading cluster ID for follower catalog manager: success
I20260812 19:59:20.014756 28403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:20.015249 28403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:20.018898 28354 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: acquiring CA information for follower catalog manager: success
I20260812 19:59:20.019137 28403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.020358 28403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.021749 28354 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:20.027127 28403 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 19:59:20.031173 28417 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.032918 28403 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:20.033181 28403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.033435 28403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:20.039577 28371 catalog_manager.cc:1295] Loaded cluster ID: 2bd2b77dccea4273857b1184baf6b455
I20260812 19:59:20.039691 28371 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: loading cluster ID for follower catalog manager: success
I20260812 19:59:20.043390 28371 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: acquiring CA information for follower catalog manager: success
I20260812 19:59:20.045866 28371 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:20.056389 28403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:20.057511 28403 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:20.057844 28403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:20.058867 28403 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:20.060622 28403 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:20.060715 28403 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.060798 28403 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:20.060846 28403 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.099365 28403 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35277
I20260812 19:59:20.099426 28545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35277 every 8 connection(s)
I20260812 19:59:20.100998 28403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:20.105696 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28403
I20260812 19:59:20.105954 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:59:20.129639 28547 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34481
I20260812 19:59:20.130199 28547 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:20.130910 28546 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34537
I20260812 19:59:20.131124 28546 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:20.131595 28548 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34495
I20260812 19:59:20.131688 28547 heartbeater.cc:507] Master 127.25.14.62:34481 requested a full tablet report, sending...
I20260812 19:59:20.131747 28546 heartbeater.cc:507] Master 127.25.14.60:34537 requested a full tablet report, sending...
I20260812 19:59:20.131798 28548 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:20.132324 28548 heartbeater.cc:507] Master 127.25.14.61:34495 requested a full tablet report, sending...
I20260812 19:59:20.134147 28215 ts_manager.cc:194] Registered new tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:20.134250 28138 ts_manager.cc:194] Registered new tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:20.136264 28215 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:52493
I20260812 19:59:20.136394 28297 ts_manager.cc:194] Registered new tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:20.145437 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:20.148277 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:44655
--webserver_interface=127.25.14.59
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655 with env {}
W20260812 19:59:20.530514 28557 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:20.530900 28557 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:20.530982 28557 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:20.539764 28557 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:20.539883 28557 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:20.539943 28557 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:20.539992 28557 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:20.550696 28557 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:20.552464 28557 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:20.554414 28557 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:20.568183 28563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:20.570895 28566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:20.570895 28564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:20.571318 28557 server_base.cc:1061] running on GCE node
I20260812 19:59:20.572260 28557 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:20.572409 28557 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:20.572476 28557 hybrid_clock.cc:648] HybridClock initialized: now 1786564760572453 us; error 0 us; skew 500 ppm
I20260812 19:59:20.572790 28557 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:20.576064 28557 webserver.cc:533] Webserver started at http://127.25.14.59:33601/ using document root <none> and password file <none>
I20260812 19:59:20.576933 28557 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:20.577076 28557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:20.577495 28557 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:20.580084 28557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.580960 28557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.587332 28557 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:59:20.591353 28571 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.592991 28557 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260812 19:59:20.593217 28557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.593490 28557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:20.611613 28557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:20.612593 28557 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:20.612861 28557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:20.635661 28557 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:44655
I20260812 19:59:20.635710 28628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:44655 every 8 connection(s)
I20260812 19:59:20.637915 28557 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:20.643095 28629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:20.647280 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28557
I20260812 19:59:20.651432 28629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:20.666880 28629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:20.674211 28629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:20.680923 28629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:20.691172 28629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Bootstrap starting.
I20260812 19:59:20.694748 28629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:20.695924 28629 log.cc:826] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:20.698784 28629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: No bootstrap required, opened a new log
I20260812 19:59:20.704380 28629 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } }
I20260812 19:59:20.704795 28629 raft_consensus.cc:385] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:20.704929 28629 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c47f30f6b3453e83ca86123fdb6ff0, State: Initialized, Role: FOLLOWER
I20260812 19:59:20.705725 28629 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } }
I20260812 19:59:20.706703 28640 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } } }
I20260812 19:59:20.707074 28640 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:20.707816 28629 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:20.718667 28653 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:20.718860 28653 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:20.720715 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28403
I20260812 19:59:20.727982 28557 master_runner.cc:429] Detected that this master b9c47f30f6b3453e83ca86123fdb6ff0 is joining an existing cluster
I20260812 19:59:20.728120 28557 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 19:59:20.729655 28557 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 19:59:20.730064 28557 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 19:59:20.730438 28557 tablet_replica.cc:333] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: stopping tablet replica
I20260812 19:59:20.732163 28540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:20.732604 28557 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:44655
I20260812 19:59:20.789562 28215 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:44655: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 44655
}
attrs {
promote: true
}
}
dest_uuid: "a21389c33ea540dd8781308e71e64efb"
cas_config_opid_index: -1
I20260812 19:59:20.791947 28215 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } }
I20260812 19:59:20.796289 28148 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 19:59:20.797225 28196 consensus_peers.cc:597] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb -> Peer b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59:44655): Couldn't send request to peer b9c47f30f6b3453e83ca86123fdb6ff0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:59:20.797495 28388 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:59:20.799266 28307 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 19:59:20.800928 28388 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:59:20.812716 28669 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.815073 28148 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.818948 28669 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:44655
I20260812 19:59:20.820101 28557 master.cc:562] Master@127.25.14.59:44655 shutting down...
I20260812 19:59:20.824542 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.824805 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:20.846385 28557 master.cc:584] Master@127.25.14.59:44655 shutdown complete.
I20260812 19:59:20.848688 28557 master_runner.cc:306] Clearing existing system tablet
I20260812 19:59:20.849476 28557 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:20.850186 28557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:20.852279 28676 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.853366 28676 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:20.853926 28674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.855029 28674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:20.856099 28672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.856401 28672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:20.857036 28557 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260812 19:59:20.857479 28307 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.875221 28678 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.875877 28557 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:20.876282 28557 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.003s sys 0.015s
I20260812 19:59:20.876442 28557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.877301 28679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:20.877717 28679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:20.883108 28557 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 19:59:20.888011 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28403
I20260812 19:59:20.892349 28557 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 19:59:20.892592 28557 log.cc:1199] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 19:59:20.893702 28557 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Deleting consensus metadata
I20260812 19:59:20.894038 28557 master_runner.cc:316] Copying system tablet from 127.25.14.61:34495
I20260812 19:59:20.903663 28557 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.61:34495
I20260812 19:59:20.913138 28238 tablet_copy_service.cc:140] P a21389c33ea540dd8781308e71e64efb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b9c47f30f6b3453e83ca86123fdb6ff0 ({username='slave'} at 127.0.0.1:53932)
I20260812 19:59:20.913492 28238 tablet_copy_service.cc:161] P a21389c33ea540dd8781308e71e64efb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b9c47f30f6b3453e83ca86123fdb6ff0 at {username='slave'} at 127.0.0.1:53932: session id = b9c47f30f6b3453e83ca86123fdb6ff0-00000000000000000000000000000000
I20260812 19:59:20.917084 28238 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:59:20.919620 28557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 19:59:20.925163 28557 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: tablet copy: Starting download of 0 data blocks...
I20260812 19:59:20.925558 28557 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: tablet copy: Starting download of 1 WAL segments...
I20260812 19:59:20.927927 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35277
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--webserver_port=44297
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655 with env {}
I20260812 19:59:20.929023 28557 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:59:20.935182 28238 tablet_copy_service.cc:342] P a21389c33ea540dd8781308e71e64efb: Request end of tablet copy session b9c47f30f6b3453e83ca86123fdb6ff0-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:53932
I20260812 19:59:20.935702 28238 tablet_copy_service.cc:434] P a21389c33ea540dd8781308e71e64efb: ending tablet copy session b9c47f30f6b3453e83ca86123fdb6ff0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b9c47f30f6b3453e83ca86123fdb6ff0
I20260812 19:59:20.957275 28557 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:20.960505 28687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:20.963040 28688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:20.974326 28557 server_base.cc:1061] running on GCE node
W20260812 19:59:20.974778 28690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:20.975428 28557 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:20.975565 28557 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:20.975648 28557 hybrid_clock.cc:648] HybridClock initialized: now 1786564760975619 us; error 0 us; skew 500 ppm
I20260812 19:59:20.976011 28557 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:20.977952 28557 webserver.cc:533] Webserver started at http://127.25.14.59:34865/ using document root <none> and password file <none>
I20260812 19:59:20.978449 28557 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:20.978559 28557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:20.982514 28557 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:59:20.984678 28695 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:20.985153 28557 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 19:59:20.985319 28557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "b9c47f30f6b3453e83ca86123fdb6ff0"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:20.985493 28557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:21.031867 28557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:21.033111 28557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:21.068961 28557 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:44655
I20260812 19:59:21.068988 28758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:44655 every 8 connection(s)
I20260812 19:59:21.070077 28557 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:21.077841 28759 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:21.080478 28759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Bootstrap starting.
I20260812 19:59:21.095629 28759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:21.096195 28759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: Bootstrap complete.
I20260812 19:59:21.097299 28759 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } }
I20260812 19:59:21.097579 28759 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9c47f30f6b3453e83ca86123fdb6ff0, State: Initialized, Role: LEARNER
I20260812 19:59:21.097918 28759 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } }
I20260812 19:59:21.098950 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:21.099327 28759 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:21.099483 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: This master's current role is: LEARNER
I20260812 19:59:21.112874 28773 catalog_manager.cc:1295] Loaded cluster ID: 2bd2b77dccea4273857b1184baf6b455
I20260812 19:59:21.113019 28773 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: loading cluster ID for follower catalog manager: success
I20260812 19:59:21.117182 28773 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: acquiring CA information for follower catalog manager: success
I20260812 19:59:21.120709 28773 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:21.248324 28727 raft_consensus.cc:1217] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 19:59:21.249315 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21389c33ea540dd8781308e71e64efb. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:21.249655 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: This master's current role is: LEARNER
W20260812 19:59:21.390600 28686 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:21.390936 28686 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:21.391021 28686 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:21.399844 28686 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:21.400076 28686 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:21.411834 28686 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44297
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:21.413913 28686 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:21.415899 28686 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:21.427690 28779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:21.427680 28780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:21.428499 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:21.429559 28686 server_base.cc:1061] running on GCE node
I20260812 19:59:21.430346 28686 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:21.430503 28686 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:21.430577 28686 hybrid_clock.cc:648] HybridClock initialized: now 1786564761430557 us; error 0 us; skew 500 ppm
I20260812 19:59:21.430977 28686 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:21.433507 28686 webserver.cc:533] Webserver started at http://127.25.14.1:44297/ using document root <none> and password file <none>
I20260812 19:59:21.434429 28686 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:21.434602 28686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:21.440963 28686 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:59:21.444331 28788 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:21.446059 28686 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:21.446273 28686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:21.447108 28686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:21.475605 28686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:21.476682 28686 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:21.477028 28686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:21.478174 28686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:21.480046 28686 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:21.480160 28686 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:21.480324 28686 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:21.480402 28686 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:21.518332 28686 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35277
I20260812 19:59:21.518385 28911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35277 every 8 connection(s)
I20260812 19:59:21.520097 28686 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:21.523283 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28686
I20260812 19:59:21.553265 28912 heartbeater.cc:344] Connected to a master server at 127.25.14.59:44655
I20260812 19:59:21.553783 28912 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:21.554843 28912 heartbeater.cc:507] Master 127.25.14.59:44655 requested a full tablet report, sending...
I20260812 19:59:21.557437 28716 ts_manager.cc:194] Registered new tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:21.560592 28913 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34537
I20260812 19:59:21.560786 28913 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:21.561512 28913 heartbeater.cc:507] Master 127.25.14.60:34537 requested a full tablet report, sending...
I20260812 19:59:21.561522 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:21.563150 28297 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:21.565503 28915 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34495
I20260812 19:59:21.565713 28915 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:21.566370 28915 heartbeater.cc:507] Master 127.25.14.61:34495 requested a full tablet report, sending...
I20260812 19:59:21.568018 28215 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:21.569067 28215 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:55389
I20260812 19:59:21.569564 28914 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34481
I20260812 19:59:21.569789 28914 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:21.570313 28914 heartbeater.cc:507] Master 127.25.14.62:34481 requested a full tablet report, sending...
I20260812 19:59:21.571765 28138 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:21.586133 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:21.586334 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28107
I20260812 19:59:21.587530 28172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:21.671869 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:21.672312 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:21.674139 28388 raft_consensus.cc:1064] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb: attempting to promote NON_VOTER b9c47f30f6b3453e83ca86123fdb6ff0 to VOTER
I20260812 19:59:21.675174 28388 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } }
I20260812 19:59:21.677812 28727 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 1 LEARNER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:21.678566 28307 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:21.678581 28927 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:21.679334 28928 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:21.688961 28927 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.690073 28727 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.693487 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.693763 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:21.694125 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.694301 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:21.694855 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.694782 28307 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.695140 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c47f30f6b3453e83ca86123fdb6ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:21.695618 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.695834 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:21.699182 28933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:21.699529 28933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404a0d960084969a07548b0477d58c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:21.706274 28148 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Refusing update from remote peer a21389c33ea540dd8781308e71e64efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:21.707019 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28107
I20260812 19:59:21.707065 28927 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 19:59:21.736460 28200 connection.cc:573] client connection to 127.25.14.62:34481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:21.737409 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34481
--webserver_interface=127.25.14.62
--webserver_port=37531
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655 with env {}
W20260812 19:59:21.738459 28200 consensus_peers.cc:597] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb -> Peer 46e5719344eb452085fe80ccfecd51e3 (127.25.14.62:34481): Couldn't send request to peer 46e5719344eb452085fe80ccfecd51e3. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:34481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:59:22.138513 28935 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:22.138998 28935 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:22.139158 28935 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:22.148304 28935 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:22.148470 28935 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:22.148595 28935 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:22.148665 28935 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:22.160285 28935 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=37531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:22.162133 28935 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:22.164112 28935 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:22.173589 28941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:22.173659 28940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:22.174850 28943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:22.175606 28935 server_base.cc:1061] running on GCE node
I20260812 19:59:22.176529 28935 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:22.176692 28935 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:22.176764 28935 hybrid_clock.cc:648] HybridClock initialized: now 1786564762176744 us; error 0 us; skew 500 ppm
I20260812 19:59:22.177095 28935 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:22.180169 28935 webserver.cc:533] Webserver started at http://127.25.14.62:37531/ using document root <none> and password file <none>
I20260812 19:59:22.180977 28935 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:22.181095 28935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:22.187381 28935 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 19:59:22.190682 28948 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:22.192423 28935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 19:59:22.192669 28935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "46e5719344eb452085fe80ccfecd51e3"
format_stamp: "Formatted at 2026-08-12 19:59:17 on dist-test-slave-p3t5"
I20260812 19:59:22.193579 28935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:22.212170 28935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:22.213294 28935 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:22.213697 28935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:22.236934 29002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34481 every 8 connection(s)
I20260812 19:59:22.236931 28935 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34481
I20260812 19:59:22.239035 28935 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:22.247184 29003 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:22.247570 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 28935
I20260812 19:59:22.247941 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28686
I20260812 19:59:22.252327 29003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Bootstrap starting.
I20260812 19:59:22.258621 28907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:22.273146 29003 log.cc:826] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:22.284838 29003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:22.285493 29003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: Bootstrap complete.
I20260812 19:59:22.293438 29003 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } }
I20260812 19:59:22.294349 29003 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e5719344eb452085fe80ccfecd51e3, State: Initialized, Role: FOLLOWER
I20260812 19:59:22.295184 29003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } }
I20260812 19:59:22.296176 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: true } } }
I20260812 19:59:22.296522 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:22.297451 29003 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:22.309774 29021 catalog_manager.cc:1295] Loaded cluster ID: 2bd2b77dccea4273857b1184baf6b455
I20260812 19:59:22.310019 29021 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: loading cluster ID for follower catalog manager: success
I20260812 19:59:22.316449 29021 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: acquiring CA information for follower catalog manager: success
I20260812 19:59:22.319659 29021 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:22.370050 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28686
I20260812 19:59:22.400017 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35277
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--webserver_port=44297
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655 with env {}
I20260812 19:59:22.566846 28976 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b9c47f30f6b3453e83ca86123fdb6ff0 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:22.575531 29031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:22.576224 29031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:22.576433 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21389c33ea540dd8781308e71e64efb. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:22.576668 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e5719344eb452085fe80ccfecd51e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:22.805173 29029 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:22.805583 29029 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:22.805745 29029 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:22.814532 29029 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:22.814729 29029 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:22.825181 29029 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44297
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:22.827054 29029 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:22.828956 29029 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:22.839000 29037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:22.840199 29038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:22.840588 29040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:22.841588 29029 server_base.cc:1061] running on GCE node
I20260812 19:59:22.842396 29029 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:22.842602 29029 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:22.842715 29029 hybrid_clock.cc:648] HybridClock initialized: now 1786564762842674 us; error 0 us; skew 500 ppm
I20260812 19:59:22.843089 29029 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:22.845599 29029 webserver.cc:533] Webserver started at http://127.25.14.1:44297/ using document root <none> and password file <none>
I20260812 19:59:22.846525 29029 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:22.846649 29029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:22.852844 29029 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 19:59:22.855975 29046 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:22.857539 29029 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:59:22.857790 29029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:22.858547 29029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:22.878261 29029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:22.879293 29029 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:22.879616 29029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:22.880738 29029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:22.882689 29029 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:22.882771 29029 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:22.882901 29029 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:22.882967 29029 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:22.922088 29029 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35277
I20260812 19:59:22.922137 29167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35277 every 8 connection(s)
I20260812 19:59:22.923744 29029 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:22.934087 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29029
I20260812 19:59:22.934502 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29029
I20260812 19:59:22.965287 29168 heartbeater.cc:344] Connected to a master server at 127.25.14.59:44655
I20260812 19:59:22.965764 29168 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:22.967460 29168 heartbeater.cc:507] Master 127.25.14.59:44655 requested a full tablet report, sending...
I20260812 19:59:22.969955 29163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:22.970163 28716 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:23.061607 29171 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34481
I20260812 19:59:23.061932 29171 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:23.062729 29171 heartbeater.cc:507] Master 127.25.14.62:34481 requested a full tablet report, sending...
I20260812 19:59:23.063416 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29029
I20260812 19:59:23.066285 28966 ts_manager.cc:194] Registered new tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
W20260812 19:59:23.084520 28950 connection.cc:573] server connection from 127.25.14.1:57715 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:23.085345 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35277
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--webserver_port=44297
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--fault_before_start=1.0 with env {}
W20260812 19:59:23.085498 29183 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:22.955019 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:56975 (local address 127.25.14.61:34495)
0812 19:59:22.970824 (+ 15805us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:22.970834 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:22.970874 (+ 40us) server_negotiation.cc:408] Connection header received
0812 19:59:22.970957 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:22.970967 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:22.971077 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:22.972243 (+ 1166us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:23.063192 (+ 90949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:23.064480 (+ 1288us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:23.084335 (+ 19855us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:56975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15630,"thread_start_us":11345,"threads_started":1}
I20260812 19:59:23.151360 28927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21389c33ea540dd8781308e71e64efb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "46e5719344eb452085fe80ccfecd51e3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34481 } } peers { permanent_uuid: "a21389c33ea540dd8781308e71e64efb" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34495 } } peers { permanent_uuid: "6404a0d960084969a07548b0477d58c6" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34537 } } peers { permanent_uuid: "b9c47f30f6b3453e83ca86123fdb6ff0" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 44655 } attrs { promote: false } } }
I20260812 19:59:23.151780 28927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21389c33ea540dd8781308e71e64efb [sys.catalog]: This master's current role is: LEADER
W20260812 19:59:23.469436 29186 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:23.469897 29186 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:23.470042 29186 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:59:23.470134 29186 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:23.479048 29186 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:23.479312 29186 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:23.490064 29186 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44297
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:23.491933 29186 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:23.494174 29186 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:23.505092 29197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:23.505062 29196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:23.505882 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:23.506310 29186 server_base.cc:1061] running on GCE node
I20260812 19:59:23.507004 29186 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:23.507230 29186 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:23.507359 29186 hybrid_clock.cc:648] HybridClock initialized: now 1786564763507300 us; error 0 us; skew 500 ppm
I20260812 19:59:23.507706 29186 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:23.510174 29186 webserver.cc:533] Webserver started at http://127.25.14.1:44297/ using document root <none> and password file <none>
I20260812 19:59:23.511008 29186 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:23.511160 29186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:23.517096 29186 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:59:23.520152 29204 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:23.521844 29186 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:59:23.522055 29186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:23.522852 29186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:23.545511 29186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:23.546628 29186 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:23.546965 29186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:23.548041 29186 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:23.550007 29186 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:23.550134 29186 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:23.550271 29186 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:23.550361 29186 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:59:23.550750 29186 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:59:23.550858 29186 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:59:23.550963 29186 fault_injection.cc:59] injecting fault for kudu (pid 29186): FLAGS_fault_before_start (process will exit)
I20260812 19:59:23.575476 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:35277
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537
--webserver_port=44297
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655 with env {}
W20260812 19:59:23.963150 29213 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:23.963490 29213 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:23.963656 29213 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:23.972216 29213 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:23.972404 29213 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:23.983417 29213 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:35277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44297
--tserver_master_addrs=127.25.14.62:34481,127.25.14.61:34495,127.25.14.60:34537,127.25.14.59:44655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:23.985201 29213 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:23.987370 29213 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:23.998481 29218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:23.998564 29219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:23.998579 29221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:23.999490 29213 server_base.cc:1061] running on GCE node
I20260812 19:59:24.000263 29213 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:24.000464 29213 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:24.000592 29213 hybrid_clock.cc:648] HybridClock initialized: now 1786564764000547 us; error 0 us; skew 500 ppm
I20260812 19:59:24.001012 29213 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:24.003584 29213 webserver.cc:533] Webserver started at http://127.25.14.1:44297/ using document root <none> and password file <none>
I20260812 19:59:24.004556 29213 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:24.004700 29213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:24.010948 29213 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:24.014087 29226 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:24.015643 29213 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 19:59:24.015856 29213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cb1e09472e19417392b26504bfc1bc38"
format_stamp: "Formatted at 2026-08-12 19:59:20 on dist-test-slave-p3t5"
I20260812 19:59:24.016685 29213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:24.042949 29213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:24.043962 29213 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:24.044351 29213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:24.045413 29213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:24.047403 29213 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:24.047484 29213 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:24.047575 29213 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:24.047621 29213 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:24.086155 29213 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35277
I20260812 19:59:24.086211 29352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35277 every 8 connection(s)
I20260812 19:59:24.087981 29213 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:24.095858 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29213
I20260812 19:59:24.096349 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29213
I20260812 19:59:24.134356 29355 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34481
I20260812 19:59:24.134887 29355 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:24.136224 29353 heartbeater.cc:344] Connected to a master server at 127.25.14.59:44655
I20260812 19:59:24.136375 29353 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:24.136502 29355 heartbeater.cc:507] Master 127.25.14.62:34481 requested a full tablet report, sending...
I20260812 19:59:24.136955 29353 heartbeater.cc:507] Master 127.25.14.59:44655 requested a full tablet report, sending...
I20260812 19:59:24.138835 28716 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:24.138924 28966 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:24.139616 29354 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34537
I20260812 19:59:24.139791 29356 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34495
I20260812 19:59:24.139817 29354 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:24.139995 29356 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:24.140615 29354 heartbeater.cc:507] Master 127.25.14.60:34537 requested a full tablet report, sending...
I20260812 19:59:24.140659 29356 heartbeater.cc:507] Master 127.25.14.61:34495 requested a full tablet report, sending...
I20260812 19:59:24.141798 28215 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:24.142151 28297 ts_manager.cc:194] Re-registered known tserver with Master: cb1e09472e19417392b26504bfc1bc38 (127.25.14.1:35277)
I20260812 19:59:24.142858 28215 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:49075
I20260812 19:59:24.143100 29348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:24.234237 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29213
I20260812 19:59:24.264264 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28935
I20260812 19:59:24.271121 28998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:24.381824 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28935
I20260812 19:59:24.413268 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28181
I20260812 19:59:24.414762 28250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:24.527841 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28181
I20260812 19:59:24.555593 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28259
I20260812 19:59:24.556998 28332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:24.683549 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28259
I20260812 19:59:24.719761 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 28557
I20260812 19:59:24.721408 28751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:24.850448 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 28557
[ OK ] ExternalMiniClusterTest.TestAddMaster/0 (7511 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/1
2026-08-12T19:59:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:24Z Disabled control of system clock
I20260812 19:59:24.953723 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40805
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981 with env {}
W20260812 19:59:25.334537 29377 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:25.334921 29377 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:25.335070 29377 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:25.344009 29377 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:25.344129 29377 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:25.344188 29377 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:25.344300 29377 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:25.355405 29377 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:25.357349 29377 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:25.359550 29377 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:25.368993 29382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:25.369005 29383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:25.370154 29385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:25.370453 29377 server_base.cc:1061] running on GCE node
I20260812 19:59:25.371400 29377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:25.373075 29377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:25.374315 29377 hybrid_clock.cc:648] HybridClock initialized: now 1786564765374254 us; error 45 us; skew 500 ppm
I20260812 19:59:25.374750 29377 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:25.377347 29377 webserver.cc:533] Webserver started at http://127.25.14.62:35767/ using document root <none> and password file <none>
I20260812 19:59:25.378278 29377 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:25.378407 29377 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:25.378834 29377 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:25.381486 29377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "aee536b895304dbd831823bc211d1787"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.382433 29377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "aee536b895304dbd831823bc211d1787"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.388652 29377 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:59:25.392724 29392 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:25.394459 29377 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:59:25.394682 29377 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "aee536b895304dbd831823bc211d1787"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.394946 29377 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:25.426826 29377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:25.427896 29377 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:25.428231 29377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:25.451213 29377 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40805
I20260812 19:59:25.451234 29447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40805 every 8 connection(s)
I20260812 19:59:25.453019 29377 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:25.453564 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29377
I20260812 19:59:25.453850 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:59:25.455140 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35805
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981 with env {}
I20260812 19:59:25.458511 29448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:25.465121 29448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:25.483397 29448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:25.484818 29396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
W20260812 19:59:25.486768 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35805: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
I20260812 19:59:25.536257 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } attempt: 1
W20260812 19:59:25.538774 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35805: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
I20260812 19:59:25.607131 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } attempt: 2
W20260812 19:59:25.609836 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35805: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
I20260812 19:59:25.701172 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } attempt: 3
W20260812 19:59:25.703660 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35805: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
W20260812 19:59:25.846534 29452 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:25.846951 29452 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:25.847107 29452 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:59:25.847016 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } attempt: 4
W20260812 19:59:25.849597 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:35805: Network error: Client connection negotiation failed: client connection to 127.25.14.61:35805: connect: Connection refused (error 111)
W20260812 19:59:25.856287 29452 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:25.856391 29452 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:25.856494 29452 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:25.856583 29452 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:25.867343 29452 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:25.869128 29452 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:25.871186 29452 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:25.881233 29458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:25.881289 29459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:25.881443 29461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:25.882299 29452 server_base.cc:1061] running on GCE node
I20260812 19:59:25.883183 29452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:25.884835 29452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:25.886102 29452 hybrid_clock.cc:648] HybridClock initialized: now 1786564765886028 us; error 66 us; skew 500 ppm
I20260812 19:59:25.886521 29452 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:25.889012 29452 webserver.cc:533] Webserver started at http://127.25.14.61:44753/ using document root <none> and password file <none>
I20260812 19:59:25.889950 29452 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:25.890101 29452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:25.890494 29452 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:25.893126 29452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "3e0b346dd057481089b0f85c2652db7d"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.894119 29452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "3e0b346dd057481089b0f85c2652db7d"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.900357 29452 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260812 19:59:25.904608 29467 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:25.906490 29452 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 19:59:25.906746 29452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "3e0b346dd057481089b0f85c2652db7d"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:25.907025 29452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:25.925063 29452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:25.926213 29452 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:25.926546 29452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:25.949457 29452 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:35805
I20260812 19:59:25.949470 29524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:35805 every 8 connection(s)
I20260812 19:59:25.951246 29452 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:59:25.956115 29525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:25.956859 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29452
I20260812 19:59:25.957062 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:59:25.958207 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41981
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981 with env {}
I20260812 19:59:25.962941 29525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:25.983999 29525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:25.995208 29525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:25.996549 29473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
W20260812 19:59:25.998801 29525 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.048178 29525 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 1
W20260812 19:59:26.050801 29525 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.119061 29525 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 2
W20260812 19:59:26.121506 29525 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.149032 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } attempt: 5
I20260812 19:59:26.156909 29448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:26.159255 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.210520 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 1
I20260812 19:59:26.212800 29525 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 3
W20260812 19:59:26.213258 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
W20260812 19:59:26.215070 29525 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.281517 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 2
W20260812 19:59:26.284011 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
W20260812 19:59:26.348342 29528 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:26.348799 29528 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:26.348951 29528 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:26.357678 29528 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:26.357851 29528 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:26.357954 29528 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:26.358042 29528 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:26.358320 29525 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 4
W20260812 19:59:26.360792 29525 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.369114 29528 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:26.370944 29528 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:26.372898 29528 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:26.381654 29536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:26.381695 29537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:26.382920 29539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:26.383360 29528 server_base.cc:1061] running on GCE node
I20260812 19:59:26.384095 29528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:26.385511 29528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:26.386750 29528 hybrid_clock.cc:648] HybridClock initialized: now 1786564766386682 us; error 50 us; skew 500 ppm
I20260812 19:59:26.387171 29528 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:26.389978 29528 webserver.cc:533] Webserver started at http://127.25.14.60:41549/ using document root <none> and password file <none>
I20260812 19:59:26.390291 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 3
I20260812 19:59:26.391000 29528 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:26.391140 29528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:26.391628 29528 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
W20260812 19:59:26.392707 29448 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:41981: Network error: Client connection negotiation failed: client connection to 127.25.14.60:41981: connect: Connection refused (error 111)
I20260812 19:59:26.394613 29528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "d29c59f60fc84502abe4697b44d0a409"
format_stamp: "Formatted at 2026-08-12 19:59:26 on dist-test-slave-p3t5"
I20260812 19:59:26.395473 29528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "d29c59f60fc84502abe4697b44d0a409"
format_stamp: "Formatted at 2026-08-12 19:59:26 on dist-test-slave-p3t5"
I20260812 19:59:26.401965 29528 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 19:59:26.405977 29547 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:26.407565 29528 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:59:26.407788 29528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "d29c59f60fc84502abe4697b44d0a409"
format_stamp: "Formatted at 2026-08-12 19:59:26 on dist-test-slave-p3t5"
I20260812 19:59:26.408093 29528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:26.438122 29528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:26.439162 29528 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:26.439482 29528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:26.463284 29528 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:41981
I20260812 19:59:26.463304 29605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:41981 every 8 connection(s)
I20260812 19:59:26.465065 29528 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:59:26.469852 29606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:26.471640 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29528
I20260812 19:59:26.471904 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:59:26.478614 29606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:26.493003 29606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:26.499760 29606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:26.509949 29606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: Bootstrap starting.
I20260812 19:59:26.513331 29606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:26.514855 29606 log.cc:826] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:26.517460 29606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: No bootstrap required, opened a new log
I20260812 19:59:26.522930 29606 raft_consensus.cc:359] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.523286 29606 raft_consensus.cc:385] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:26.523448 29606 raft_consensus.cc:740] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29c59f60fc84502abe4697b44d0a409, State: Initialized, Role: FOLLOWER
I20260812 19:59:26.524081 29606 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.525040 29612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.525283 29612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.526190 29606 sys_catalog.cc:565] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:26.536525 29625 catalog_manager.cc:1594] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:26.536690 29625 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:26.534217 29528 master_runner.cc:187] Error getting master registration for 127.25.14.62:40805: OK, instance_id { permanent_uuid: "aee536b895304dbd831823bc211d1787" instance_seqno: 1786564765444766 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:26.541579 29528 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:26.569921 29448 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 4
I20260812 19:59:26.581141 29448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Bootstrap starting.
I20260812 19:59:26.584533 29448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:26.585740 29448 log.cc:826] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:26.588413 29448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: No bootstrap required, opened a new log
I20260812 19:59:26.593834 29448 raft_consensus.cc:359] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.594221 29448 raft_consensus.cc:385] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:26.594427 29448 raft_consensus.cc:740] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee536b895304dbd831823bc211d1787, State: Initialized, Role: FOLLOWER
I20260812 19:59:26.595252 29448 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.596438 29628 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.596751 29628 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.597704 29448 sys_catalog.cc:565] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:26.609511 29640 catalog_manager.cc:1594] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:26.609756 29640 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:26.609361 29377 master_runner.cc:187] Error getting master registration for 127.25.14.61:35805: OK, instance_id { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" instance_seqno: 1786564765943334 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:26.615829 29377 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:26.660049 29525 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } attempt: 5
I20260812 19:59:26.673027 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: Bootstrap starting.
I20260812 19:59:26.676777 29525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:26.678089 29525 log.cc:826] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:26.681218 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: No bootstrap required, opened a new log
W20260812 19:59:26.681730 29398 tablet.cc:2406] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:26.689426 29525 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.690092 29525 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:26.690294 29525 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0b346dd057481089b0f85c2652db7d, State: Initialized, Role: FOLLOWER
I20260812 19:59:26.691251 29525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
W20260812 19:59:26.691843 29554 tablet.cc:2406] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:26.693975 29644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.694335 29644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.695487 29525 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:26.713969 29452 master_runner.cc:264] No leader master found from master 3e0b346dd057481089b0f85c2652db7d
I20260812 19:59:26.714146 29452 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:59:26.720199 29655 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:26.720501 29655 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:26.777328 29644 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:59:26.777796 29644 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.779970 29644 leader_election.cc:290] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee536b895304dbd831823bc211d1787 (127.25.14.62:40805), d29c59f60fc84502abe4697b44d0a409 (127.25.14.60:41981)
I20260812 19:59:26.781263 29422 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e0b346dd057481089b0f85c2652db7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee536b895304dbd831823bc211d1787" is_pre_election: true
I20260812 19:59:26.782344 29422 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e0b346dd057481089b0f85c2652db7d in term 0.
I20260812 19:59:26.783272 29472 leader_election.cc:304] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e0b346dd057481089b0f85c2652db7d, aee536b895304dbd831823bc211d1787; no voters:
I20260812 19:59:26.784633 29580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e0b346dd057481089b0f85c2652db7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29c59f60fc84502abe4697b44d0a409" is_pre_election: true
I20260812 19:59:26.785195 29580 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e0b346dd057481089b0f85c2652db7d in term 0.
I20260812 19:59:26.786343 29644 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:59:26.786537 29644 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:59:26.786649 29644 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:26.794970 29644 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.798686 29422 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e0b346dd057481089b0f85c2652db7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee536b895304dbd831823bc211d1787"
I20260812 19:59:26.798960 29422 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:26.803268 29422 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0b346dd057481089b0f85c2652db7d in term 1.
I20260812 19:59:26.804064 29472 leader_election.cc:304] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e0b346dd057481089b0f85c2652db7d, aee536b895304dbd831823bc211d1787; no voters:
I20260812 19:59:26.808979 29580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e0b346dd057481089b0f85c2652db7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29c59f60fc84502abe4697b44d0a409"
I20260812 19:59:26.809244 29580 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:26.813187 29580 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0b346dd057481089b0f85c2652db7d in term 1.
I20260812 19:59:26.814409 29644 leader_election.cc:290] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [CANDIDATE]: Term 1 election: Requested vote from peers aee536b895304dbd831823bc211d1787 (127.25.14.62:40805), d29c59f60fc84502abe4697b44d0a409 (127.25.14.60:41981)
I20260812 19:59:26.815004 29644 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:59:26.815567 29644 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 1 LEADER]: Becoming Leader. State: Replica: 3e0b346dd057481089b0f85c2652db7d, State: Running, Role: LEADER
I20260812 19:59:26.816429 29644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } }
I20260812 19:59:26.832670 29658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0b346dd057481089b0f85c2652db7d. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.833141 29658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:26.840268 29660 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:26.843137 29660 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:26.865554 29580 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:26.867075 29658 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:26.875031 29422 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:26.876708 29658 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:26.914122 29628 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0b346dd057481089b0f85c2652db7d. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.914484 29628 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.922214 29661 mvcc.cc:204] Tried to move back new op lower bound from 7317769285065904128 to 7317769284894609408. Current Snapshot: MvccSnapshot[applied={T|T < 7317769285065904128}]
I20260812 19:59:26.926548 29664 mvcc.cc:204] Tried to move back new op lower bound from 7317769285065904128 to 7317769284894609408. Current Snapshot: MvccSnapshot[applied={T|T < 7317769285065904128}]
I20260812 19:59:26.934545 29644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.934908 29644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:26.939095 29612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0b346dd057481089b0f85c2652db7d. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.939410 29612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.944612 29628 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.944936 29628 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:26.946959 29658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.947963 29658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:26.955701 29660 catalog_manager.cc:1383] Generated new cluster ID: 2ea952de77a84ae99b6f4eea2ea15b53
I20260812 19:59:26.955825 29660 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:26.973807 29663 mvcc.cc:204] Tried to move back new op lower bound from 7317769285065904128 to 7317769284894609408. Current Snapshot: MvccSnapshot[applied={T|T < 7317769285065904128}]
I20260812 19:59:26.986143 29612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } }
I20260812 19:59:26.986466 29612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:27.008824 29660 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:59:27.010836 29660 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:27.037240 29660 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: Generated new TSK 0
I20260812 19:59:27.038653 29660 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:59:27.056648 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:59:27.500092 29672 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:27.500488 29672 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:27.500625 29672 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:27.509471 29672 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:27.509732 29672 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:27.520790 29672 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:27.522706 29672 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:27.524727 29672 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:27.536053 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:27.536778 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:27.537125 29681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:27.538255 29672 server_base.cc:1061] running on GCE node
I20260812 19:59:27.539094 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:27.539366 29625 catalog_manager.cc:1295] Loaded cluster ID: 2ea952de77a84ae99b6f4eea2ea15b53
I20260812 19:59:27.539466 29625 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: loading cluster ID for follower catalog manager: success
I20260812 19:59:27.540452 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:27.541774 29672 hybrid_clock.cc:648] HybridClock initialized: now 1786564767541689 us; error 63 us; skew 500 ppm
I20260812 19:59:27.542182 29672 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:27.543203 29625 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: acquiring CA information for follower catalog manager: success
I20260812 19:59:27.545399 29672 webserver.cc:533] Webserver started at http://127.25.14.1:36647/ using document root <none> and password file <none>
I20260812 19:59:27.545655 29625 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:27.546371 29672 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:27.546499 29672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:27.546978 29672 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:27.549661 29672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:27.550529 29672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:27.556381 29672 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:59:27.560158 29688 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:27.561805 29672 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 19:59:27.561998 29672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:27.562248 29672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:27.578055 29672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:27.579008 29672 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:27.579316 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:27.580447 29672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:27.582345 29672 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:27.582474 29672 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:27.582618 29672 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:27.582719 29672 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:27.612564 29640 catalog_manager.cc:1295] Loaded cluster ID: 2ea952de77a84ae99b6f4eea2ea15b53
I20260812 19:59:27.612718 29640 catalog_manager.cc:1588] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: loading cluster ID for follower catalog manager: success
I20260812 19:59:27.617328 29640 catalog_manager.cc:1610] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: acquiring CA information for follower catalog manager: success
I20260812 19:59:27.620285 29640 catalog_manager.cc:1638] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:27.620702 29672 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37617
I20260812 19:59:27.620862 29812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37617 every 8 connection(s)
I20260812 19:59:27.622308 29672 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:27.628474 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29672
I20260812 19:59:27.628666 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:59:27.667157 29816 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35805
I20260812 19:59:27.667658 29816 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:27.667893 29814 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40805
I20260812 19:59:27.668079 29814 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:27.669076 29816 heartbeater.cc:507] Master 127.25.14.61:35805 requested a full tablet report, sending...
I20260812 19:59:27.669358 29814 heartbeater.cc:507] Master 127.25.14.62:40805 requested a full tablet report, sending...
I20260812 19:59:27.670166 29813 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41981
I20260812 19:59:27.670303 29813 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:27.670827 29813 heartbeater.cc:507] Master 127.25.14.60:41981 requested a full tablet report, sending...
I20260812 19:59:27.672005 29412 ts_manager.cc:194] Registered new tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:27.672049 29487 ts_manager.cc:194] Registered new tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:27.672786 29567 ts_manager.cc:194] Registered new tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:27.674305 29487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:38301
I20260812 19:59:27.676987 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:27.679520 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:34959
--webserver_interface=127.25.14.59
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959 with env {}
W20260812 19:59:28.062605 29825 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:28.063035 29825 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:28.063179 29825 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:28.072145 29825 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:28.072291 29825 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:28.072419 29825 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:28.072504 29825 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:28.083281 29825 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:28.085170 29825 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:28.087337 29825 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:28.096362 29830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:28.096998 29832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:28.098068 29834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:28.098165 29825 server_base.cc:1061] running on GCE node
I20260812 19:59:28.099180 29825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:28.100915 29825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:28.102190 29825 hybrid_clock.cc:648] HybridClock initialized: now 1786564768102132 us; error 46 us; skew 500 ppm
I20260812 19:59:28.102633 29825 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:28.105041 29825 webserver.cc:533] Webserver started at http://127.25.14.59:34953/ using document root <none> and password file <none>
I20260812 19:59:28.105959 29825 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:28.106106 29825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:28.106514 29825 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:28.109076 29825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "218f8614779047d9b8fb4099193ecccf"
format_stamp: "Formatted at 2026-08-12 19:59:28 on dist-test-slave-p3t5"
I20260812 19:59:28.110024 29825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "218f8614779047d9b8fb4099193ecccf"
format_stamp: "Formatted at 2026-08-12 19:59:28 on dist-test-slave-p3t5"
I20260812 19:59:28.116645 29825 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 19:59:28.121061 29841 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:28.122855 29825 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:28.123093 29825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "218f8614779047d9b8fb4099193ecccf"
format_stamp: "Formatted at 2026-08-12 19:59:28 on dist-test-slave-p3t5"
I20260812 19:59:28.123363 29825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:28.141570 29825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:28.142752 29825 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:28.143103 29825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:28.166519 29825 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:34959
I20260812 19:59:28.166538 29893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:34959 every 8 connection(s)
I20260812 19:59:28.168394 29825 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:28.169209 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 29825
I20260812 19:59:28.174206 29894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:28.182111 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:28.193339 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:28.200537 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:28.207636 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:28.218003 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Bootstrap starting.
I20260812 19:59:28.221410 29894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:28.222654 29894 log.cc:826] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:28.225476 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: No bootstrap required, opened a new log
I20260812 19:59:28.231293 29894 raft_consensus.cc:359] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } }
I20260812 19:59:28.231676 29894 raft_consensus.cc:385] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:28.231805 29894 raft_consensus.cc:740] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218f8614779047d9b8fb4099193ecccf, State: Initialized, Role: FOLLOWER
I20260812 19:59:28.232533 29894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } }
I20260812 19:59:28.233485 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } } }
I20260812 19:59:28.233902 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:28.234720 29894 sys_catalog.cc:565] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:28.245471 29917 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:28.245658 29917 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:28.254315 29825 master_runner.cc:429] Detected that this master 218f8614779047d9b8fb4099193ecccf is joining an existing cluster
I20260812 19:59:28.254458 29825 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 19:59:28.255555 29825 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 19:59:28.255934 29825 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 19:59:28.256179 29825 tablet_replica.cc:333] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: stopping tablet replica
I20260812 19:59:28.257903 29825 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:34959
I20260812 19:59:28.300983 29487 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:34959: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "218f8614779047d9b8fb4099193ecccf"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 34959
}
attrs {
promote: true
}
}
dest_uuid: "3e0b346dd057481089b0f85c2652db7d"
cas_config_opid_index: -1
I20260812 19:59:28.302606 29487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } }
I20260812 19:59:28.306208 29421 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 19:59:28.307173 29658 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260812 19:59:28.307567 29471 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d -> Peer 218f8614779047d9b8fb4099193ecccf (127.25.14.59:34959): Couldn't send request to peer 218f8614779047d9b8fb4099193ecccf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:59:28.309247 29580 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 19:59:28.309955 29658 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:59:28.312366 29658 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 218f8614779047d9b8fb4099193ecccf (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.314450 29421 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 218f8614779047d9b8fb4099193ecccf (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.314751 29580 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 218f8614779047d9b8fb4099193ecccf (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.326912 29935 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.327503 29935 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:28.327708 29658 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:34959
I20260812 19:59:28.328809 29825 master.cc:562] Master@127.25.14.59:34959 shutting down...
I20260812 19:59:28.330212 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.330545 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:28.333937 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.334296 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:28.334734 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.334579 29934 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.335011 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:28.335084 29934 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:28.365302 29825 master.cc:584] Master@127.25.14.59:34959 shutdown complete.
I20260812 19:59:28.365479 29825 master_runner.cc:306] Clearing existing system tablet
I20260812 19:59:28.365958 29825 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:28.366104 29825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:28.374593 29825 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260812 19:59:28.376495 29939 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:28.376940 29825 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:28.377246 29825 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 19:59:28.377386 29825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "218f8614779047d9b8fb4099193ecccf"
format_stamp: "Formatted at 2026-08-12 19:59:28 on dist-test-slave-p3t5"
I20260812 19:59:28.380942 29825 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 19:59:28.386436 29825 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 19:59:28.386632 29825 log.cc:1199] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 19:59:28.387198 29825 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Deleting consensus metadata
I20260812 19:59:28.387480 29825 master_runner.cc:316] Copying system tablet from 127.25.14.61:35805
I20260812 19:59:28.390553 29825 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.61:35805
I20260812 19:59:28.398674 29509 tablet_copy_service.cc:140] P 3e0b346dd057481089b0f85c2652db7d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 218f8614779047d9b8fb4099193ecccf ({username='slave'} at 127.0.0.1:32896)
I20260812 19:59:28.399001 29509 tablet_copy_service.cc:161] P 3e0b346dd057481089b0f85c2652db7d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 218f8614779047d9b8fb4099193ecccf at {username='slave'} at 127.0.0.1:32896: session id = 218f8614779047d9b8fb4099193ecccf-00000000000000000000000000000000
I20260812 19:59:28.402379 29509 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:59:28.404716 29825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 19:59:28.410111 29825 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: tablet copy: Starting download of 0 data blocks...
I20260812 19:59:28.413527 29825 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: tablet copy: Starting download of 1 WAL segments...
I20260812 19:59:28.417147 29825 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:59:28.421159 29509 tablet_copy_service.cc:342] P 3e0b346dd057481089b0f85c2652db7d: Request end of tablet copy session 218f8614779047d9b8fb4099193ecccf-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:32896
I20260812 19:59:28.421460 29509 tablet_copy_service.cc:434] P 3e0b346dd057481089b0f85c2652db7d: ending tablet copy session 218f8614779047d9b8fb4099193ecccf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 218f8614779047d9b8fb4099193ecccf
W20260812 19:59:28.429761 29825 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 19:59:28.433449 29825 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:28.436156 29946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:28.436828 29947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:28.438649 29949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:28.445135 29825 server_base.cc:1061] running on GCE node
I20260812 19:59:28.445648 29825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:28.446403 29825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:28.447638 29825 hybrid_clock.cc:648] HybridClock initialized: now 1786564768447583 us; error 36 us; skew 500 ppm
I20260812 19:59:28.447966 29825 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:28.449703 29825 webserver.cc:533] Webserver started at http://127.25.14.59:40485/ using document root <none> and password file <none>
I20260812 19:59:28.450100 29825 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:28.450222 29825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:28.453333 29825 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 19:59:28.455502 29956 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:28.455981 29825 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 19:59:28.456117 29825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "218f8614779047d9b8fb4099193ecccf"
format_stamp: "Formatted at 2026-08-12 19:59:28 on dist-test-slave-p3t5"
I20260812 19:59:28.456284 29825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:28.480902 29825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:28.481768 29825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:28.504558 29825 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:34959
I20260812 19:59:28.504613 30013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:34959 every 8 connection(s)
I20260812 19:59:28.505528 29825 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:28.509155 30014 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:28.511600 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Bootstrap starting.
I20260812 19:59:28.528618 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:28.529234 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: Bootstrap complete.
I20260812 19:59:28.530444 30014 raft_consensus.cc:359] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } }
I20260812 19:59:28.530653 30014 raft_consensus.cc:740] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 218f8614779047d9b8fb4099193ecccf, State: Initialized, Role: LEARNER
I20260812 19:59:28.530923 30014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } }
I20260812 19:59:28.531944 30014 sys_catalog.cc:565] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:28.531862 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.532202 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: This master's current role is: LEARNER
I20260812 19:59:28.541242 30029 catalog_manager.cc:1295] Loaded cluster ID: 2ea952de77a84ae99b6f4eea2ea15b53
I20260812 19:59:28.541361 30029 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: loading cluster ID for follower catalog manager: success
I20260812 19:59:28.545037 30029 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: acquiring CA information for follower catalog manager: success
I20260812 19:59:28.547689 30029 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:28.580015 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29672
I20260812 19:59:28.589319 29806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:28.678929 29816 heartbeater.cc:499] Master 127.25.14.61:35805 was elected leader, sending a full tablet report...
I20260812 19:59:28.686950 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29672
I20260812 19:59:28.719921 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37617
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--webserver_port=36647
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959 with env {}
I20260812 19:59:28.823109 29986 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 19:59:28.823895 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0b346dd057481089b0f85c2652db7d. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:28.824199 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: This master's current role is: LEARNER
W20260812 19:59:29.122510 30032 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:29.122844 30032 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:29.123042 30032 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:29.131789 30032 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:29.132025 30032 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:29.143704 30032 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37617
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36647
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:29.145546 30032 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:29.147639 30032 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:29.158389 30040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:29.158318 30039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:29.159410 30042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:29.159695 30032 server_base.cc:1061] running on GCE node
I20260812 19:59:29.160414 30032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:29.161423 30032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:29.162660 30032 hybrid_clock.cc:648] HybridClock initialized: now 1786564769162593 us; error 64 us; skew 500 ppm
I20260812 19:59:29.163009 30032 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:29.165304 30032 webserver.cc:533] Webserver started at http://127.25.14.1:36647/ using document root <none> and password file <none>
I20260812 19:59:29.166190 30032 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:29.166349 30032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:29.172217 30032 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 19:59:29.175251 30048 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:29.176903 30032 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:29.177103 30032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:29.177948 30032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:29.205746 30032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:29.206765 30032 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:29.207098 30032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:29.208168 30032 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:29.210210 30032 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:29.210299 30032 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:29.210434 30032 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:29.210495 30032 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:29.246618 30032 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37617
I20260812 19:59:29.246665 30177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37617 every 8 connection(s)
I20260812 19:59:29.248342 30032 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:29.251037 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30032
I20260812 19:59:29.285243 30180 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40805
I20260812 19:59:29.285415 30178 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41981
I20260812 19:59:29.285719 30180 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:29.285750 30178 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:29.287094 30178 heartbeater.cc:507] Master 127.25.14.60:41981 requested a full tablet report, sending...
I20260812 19:59:29.287127 30181 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35805
I20260812 19:59:29.287302 30181 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:29.287801 30181 heartbeater.cc:507] Master 127.25.14.61:35805 requested a full tablet report, sending...
I20260812 19:59:29.288072 30180 heartbeater.cc:507] Master 127.25.14.62:40805 requested a full tablet report, sending...
I20260812 19:59:29.289285 29487 ts_manager.cc:194] Re-registered known tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:29.289434 29412 ts_manager.cc:194] Re-registered known tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:29.289451 29566 ts_manager.cc:194] Re-registered known tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:29.290123 30179 heartbeater.cc:344] Connected to a master server at 127.25.14.59:34959
I20260812 19:59:29.290302 30179 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:29.290490 29487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:43321
I20260812 19:59:29.290866 30179 heartbeater.cc:507] Master 127.25.14.59:34959 requested a full tablet report, sending...
I20260812 19:59:29.292764 29976 ts_manager.cc:194] Registered new tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:29.294003 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:29.303814 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:29.304030 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29377
I20260812 19:59:29.305131 29442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:29.372589 30192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:29.372893 30192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:29.373905 30191 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d: attempting to promote NON_VOTER 218f8614779047d9b8fb4099193ecccf to VOTER
I20260812 19:59:29.375379 30191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } }
I20260812 19:59:29.378779 29986 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 1 LEARNER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:29.378783 29580 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:29.379791 30192 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:29.380237 30193 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:29.390280 30196 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 218f8614779047d9b8fb4099193ecccf (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.391219 29580 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 218f8614779047d9b8fb4099193ecccf (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.393857 30191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.394097 30191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:29.394508 30191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.394694 30191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:29.395040 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.395314 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:29.396055 29986 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 218f8614779047d9b8fb4099193ecccf (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.398131 30200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.398452 30200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c59f60fc84502abe4697b44d0a409 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:29.400323 30201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:29.400662 30201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218f8614779047d9b8fb4099193ecccf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:29.428181 29421 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Refusing update from remote peer 3e0b346dd057481089b0f85c2652db7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:29.428627 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29377
I20260812 19:59:29.432392 30193 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 19:59:29.461186 29472 connection.cc:573] client connection to 127.25.14.62:40805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:29.461905 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40805
--webserver_interface=127.25.14.62
--webserver_port=35767
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959 with env {}
W20260812 19:59:29.463703 29472 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d -> Peer aee536b895304dbd831823bc211d1787 (127.25.14.62:40805): Couldn't send request to peer aee536b895304dbd831823bc211d1787. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:40805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:59:29.862495 30205 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:29.862920 30205 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:29.863062 30205 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:29.872156 30205 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:29.872300 30205 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:29.872427 30205 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:29.872512 30205 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:29.883462 30205 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:29.885442 30205 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:29.887717 30205 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:29.897848 30210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:29.899065 30213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:29.897877 30211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:29.899607 30205 server_base.cc:1061] running on GCE node
I20260812 19:59:29.900511 30205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:29.902272 30205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:29.903532 30205 hybrid_clock.cc:648] HybridClock initialized: now 1786564769903487 us; error 44 us; skew 500 ppm
I20260812 19:59:29.903941 30205 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:29.906453 30205 webserver.cc:533] Webserver started at http://127.25.14.62:35767/ using document root <none> and password file <none>
I20260812 19:59:29.907341 30205 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:29.907518 30205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:29.913990 30205 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 19:59:29.917183 30222 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:29.918843 30205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:29.919078 30205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "aee536b895304dbd831823bc211d1787"
format_stamp: "Formatted at 2026-08-12 19:59:25 on dist-test-slave-p3t5"
I20260812 19:59:29.919898 30205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:29.946494 30205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:29.947551 30205 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:29.947906 30205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:29.972232 30205 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40805
I20260812 19:59:29.972259 30275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40805 every 8 connection(s)
I20260812 19:59:29.974145 30205 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:29.981948 30276 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:29.983306 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30205
I20260812 19:59:29.983711 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30032
I20260812 19:59:29.986852 30276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Bootstrap starting.
I20260812 19:59:29.994277 30173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:30.009255 30276 log.cc:826] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:30.020463 30276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:30.021132 30276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: Bootstrap complete.
I20260812 19:59:30.028920 30276 raft_consensus.cc:359] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } }
I20260812 19:59:30.029896 30276 raft_consensus.cc:740] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee536b895304dbd831823bc211d1787, State: Initialized, Role: FOLLOWER
I20260812 19:59:30.031076 30276 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } }
I20260812 19:59:30.032145 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: true } } }
I20260812 19:59:30.032521 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:30.033577 30276 sys_catalog.cc:565] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:30.046629 30296 catalog_manager.cc:1295] Loaded cluster ID: 2ea952de77a84ae99b6f4eea2ea15b53
I20260812 19:59:30.046947 30296 catalog_manager.cc:1588] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: loading cluster ID for follower catalog manager: success
I20260812 19:59:30.056767 30296 catalog_manager.cc:1610] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: acquiring CA information for follower catalog manager: success
I20260812 19:59:30.060468 30296 catalog_manager.cc:1638] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:30.101596 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30032
I20260812 19:59:30.132822 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37617
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--webserver_port=36647
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959 with env {}
I20260812 19:59:30.270678 30250 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 218f8614779047d9b8fb4099193ecccf (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:30.280552 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0b346dd057481089b0f85c2652db7d. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:30.280901 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:30.280802 30306 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:30.281283 30306 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee536b895304dbd831823bc211d1787 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:30.538148 30304 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:30.538515 30304 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:30.538654 30304 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:30.547437 30304 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:30.547657 30304 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:30.558471 30304 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37617
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36647
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:30.560360 30304 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:30.562656 30304 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:30.575421 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:30.576184 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:30.577584 30316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:30.578047 30304 server_base.cc:1061] running on GCE node
I20260812 19:59:30.578730 30304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:30.579885 30304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:30.581141 30304 hybrid_clock.cc:648] HybridClock initialized: now 1786564770581094 us; error 59 us; skew 500 ppm
I20260812 19:59:30.581497 30304 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:30.583859 30304 webserver.cc:533] Webserver started at http://127.25.14.1:36647/ using document root <none> and password file <none>
I20260812 19:59:30.584699 30304 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:30.584856 30304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:30.590957 30304 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:30.594421 30323 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:30.595924 30304 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:59:30.596144 30304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:30.597121 30304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:30.629006 30304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:30.630300 30304 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:30.630666 30304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:30.631870 30304 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:30.634208 30304 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:30.634333 30304 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:30.634550 30304 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:30.634627 30304 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:30.675063 30304 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37617
I20260812 19:59:30.675104 30445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37617 every 8 connection(s)
I20260812 19:59:30.676749 30304 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:30.683864 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30304
I20260812 19:59:30.684194 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30304
I20260812 19:59:30.711208 30441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:30.789054 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e0b346dd057481089b0f85c2652db7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aee536b895304dbd831823bc211d1787" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40805 } } peers { permanent_uuid: "3e0b346dd057481089b0f85c2652db7d" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 35805 } } peers { permanent_uuid: "d29c59f60fc84502abe4697b44d0a409" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 41981 } } peers { permanent_uuid: "218f8614779047d9b8fb4099193ecccf" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 34959 } attrs { promote: false } } }
I20260812 19:59:30.789479 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0b346dd057481089b0f85c2652db7d [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:30.809916 30447 heartbeater.cc:344] Connected to a master server at 127.25.14.59:34959
I20260812 19:59:30.810293 30449 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40805
I20260812 19:59:30.810536 30449 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:30.810542 30447 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:30.811973 30447 heartbeater.cc:507] Master 127.25.14.59:34959 requested a full tablet report, sending...
I20260812 19:59:30.813017 30446 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41981
I20260812 19:59:30.813164 30446 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:30.813323 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30304
I20260812 19:59:30.839187 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37617
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--webserver_port=36647
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--fault_before_start=1.0 with env {}
W20260812 19:59:30.839600 30464 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:30.691768 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:56689 (local address 127.25.14.61:35805)
0812 19:59:30.707735 (+ 15967us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:30.707752 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:30.707773 (+ 21us) server_negotiation.cc:408] Connection header received
0812 19:59:30.707844 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:30.707852 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:30.707947 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:30.708059 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:30.709289 (+ 1230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:30.710663 (+ 1374us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:30.812271 (+101608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:30.812764 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 19:59:30.815367 (+ 2603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 19:59:30.815418 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 19:59:30.815427 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0812 19:59:30.815455 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0812 19:59:30.815518 (+ 63us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0812 19:59:30.815540 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0812 19:59:30.815642 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0812 19:59:30.838451 (+ 22809us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:56689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15838,"thread_start_us":16407,"threads_started":1}
W20260812 19:59:31.236056 30470 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:31.236480 30470 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:31.236630 30470 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:59:31.236740 30470 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:31.245703 30470 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:31.246031 30470 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:31.256511 30470 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37617
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36647
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:31.258512 30470 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:31.260973 30470 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:31.271358 30476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:31.271770 30477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:31.273037 30479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:31.273573 30470 server_base.cc:1061] running on GCE node
I20260812 19:59:31.274272 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:31.275317 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:31.276616 30470 hybrid_clock.cc:648] HybridClock initialized: now 1786564771276521 us; error 62 us; skew 500 ppm
I20260812 19:59:31.276986 30470 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:31.279532 30470 webserver.cc:533] Webserver started at http://127.25.14.1:36647/ using document root <none> and password file <none>
I20260812 19:59:31.280440 30470 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:31.280612 30470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:31.287047 30470 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:31.290761 30485 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:31.292730 30470 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:31.292924 30470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:31.293843 30470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:31.314785 30470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:31.315739 30470 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:31.316025 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:31.317040 30470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:31.318943 30470 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:31.319041 30470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:31.319204 30470 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:31.319304 30470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:59:31.319741 30470 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:59:31.319902 30470 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:59:31.320004 30470 fault_injection.cc:59] injecting fault for kudu (pid 30470): FLAGS_fault_before_start (process will exit)
I20260812 19:59:31.350590 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:37617
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981
--webserver_port=36647
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959 with env {}
W20260812 19:59:31.746459 30494 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:31.746838 30494 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:31.747013 30494 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:31.755910 30494 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:31.756174 30494 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:31.768399 30494 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:37617
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36647
--tserver_master_addrs=127.25.14.62:40805,127.25.14.61:35805,127.25.14.60:41981,127.25.14.59:34959
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:31.770385 30494 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:31.772538 30494 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:31.784237 30502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:31.784166 30503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:31.785400 30506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:31.785595 30494 server_base.cc:1061] running on GCE node
I20260812 19:59:31.786450 30494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:31.787593 30494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:31.788894 30494 hybrid_clock.cc:648] HybridClock initialized: now 1786564771788831 us; error 44 us; skew 500 ppm
I20260812 19:59:31.789376 30494 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:31.792315 30494 webserver.cc:533] Webserver started at http://127.25.14.1:36647/ using document root <none> and password file <none>
I20260812 19:59:31.793326 30494 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:31.793449 30494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:31.800662 30494 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:59:31.804404 30512 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:31.806192 30494 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:31.806403 30494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "edab733e101f46adbcbd6e3c3ed0a047"
format_stamp: "Formatted at 2026-08-12 19:59:27 on dist-test-slave-p3t5"
I20260812 19:59:31.807264 30494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:31.838061 30494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:31.839077 30494 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:31.839438 30494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:31.840634 30494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:31.842819 30494 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:31.842945 30494 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:31.843119 30494 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:31.843196 30494 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:31.880321 30494 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37617
I20260812 19:59:31.880393 30634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37617 every 8 connection(s)
I20260812 19:59:31.882316 30494 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:31.892181 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30494
I20260812 19:59:31.892601 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30494
I20260812 19:59:31.922415 30635 heartbeater.cc:344] Connected to a master server at 127.25.14.60:41981
I20260812 19:59:31.922940 30635 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:31.924283 30635 heartbeater.cc:507] Master 127.25.14.60:41981 requested a full tablet report, sending...
I20260812 19:59:31.924307 30639 heartbeater.cc:344] Connected to a master server at 127.25.14.61:35805
I20260812 19:59:31.924527 30639 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:31.924607 30630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:31.926568 29566 ts_manager.cc:194] Re-registered known tserver with Master: edab733e101f46adbcbd6e3c3ed0a047 (127.25.14.1:37617)
I20260812 19:59:32.015507 30638 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40805
I20260812 19:59:32.015786 30638 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:32.016057 30636 heartbeater.cc:344] Connected to a master server at 127.25.14.59:34959
I20260812 19:59:32.016228 30636 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:32.016770 30636 heartbeater.cc:507] Master 127.25.14.59:34959 requested a full tablet report, sending...
I20260812 19:59:32.016227 30639 heartbeater.cc:507] Master 127.25.14.61:35805 requested a full tablet report, sending...
I20260812 19:59:32.017012 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30494
I20260812 19:59:32.042363 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30205
I20260812 19:59:32.050307 30271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:32.162951 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30205
I20260812 19:59:32.193550 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29452
I20260812 19:59:32.194911 29520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:32.310249 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29452
W20260812 19:59:32.341531 29551 connection.cc:573] server connection from 127.0.0.1:46422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:32.341987 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29528
I20260812 19:59:32.343448 29600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:32.456223 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29528
I20260812 19:59:32.485839 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 29825
I20260812 19:59:32.487334 30009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:32.605325 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 29825
2026-08-12T19:59:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/1 (7763 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/2
2026-08-12T19:59:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:32Z Disabled control of system clock
2026-08-12T19:59:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:32Z Disabled control of system clock
2026-08-12T19:59:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:32Z Disabled control of system clock
2026-08-12T19:59:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:32Z Disabled control of system clock
2026-08-12T19:59:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T19:59:32Z Disabled control of system clock
I20260812 19:59:32.963692 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38065
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975 with env {}
W20260812 19:59:33.349908 30667 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:33.350299 30667 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:33.350373 30667 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:33.359340 30667 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:33.359470 30667 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:33.359537 30667 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:33.359578 30667 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:33.370615 30667 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:33.372443 30667 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:33.374501 30667 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:33.383857 30673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:33.384166 30675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:33.383939 30672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:33.385512 30667 server_base.cc:1061] running on GCE node
I20260812 19:59:33.386468 30667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:33.388284 30667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:33.389665 30667 hybrid_clock.cc:648] HybridClock initialized: now 1786564773389454 us; error 129 us; skew 500 ppm
I20260812 19:59:33.390074 30667 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:33.392480 30667 webserver.cc:533] Webserver started at http://127.25.14.62:46473/ using document root <none> and password file <none>
I20260812 19:59:33.393373 30667 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:33.393522 30667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:33.393981 30667 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:33.396629 30667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "7036daa927394cc2ae6c89c11056be04"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.397524 30667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "7036daa927394cc2ae6c89c11056be04"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.403923 30667 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:59:33.408377 30683 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:33.410228 30667 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:33.410485 30667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "7036daa927394cc2ae6c89c11056be04"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.410784 30667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:33.428745 30667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:33.429826 30667 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:33.430137 30667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:33.452991 30667 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38065
I20260812 19:59:33.453011 30738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38065 every 8 connection(s)
I20260812 19:59:33.455029 30667 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:33.459910 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:33.464846 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30667
I20260812 19:59:33.465121 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:59:33.466293 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43755
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975 with env {}
I20260812 19:59:33.466672 30739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:33.485033 30739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:33.486630 30686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
W20260812 19:59:33.488705 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43755: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
I20260812 19:59:33.538091 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } attempt: 1
W20260812 19:59:33.540804 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43755: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
I20260812 19:59:33.609108 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } attempt: 2
W20260812 19:59:33.611514 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43755: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
I20260812 19:59:33.702847 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } attempt: 3
W20260812 19:59:33.705452 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43755: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
I20260812 19:59:33.848791 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } attempt: 4
W20260812 19:59:33.851099 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43755: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43755: connect: Connection refused (error 111)
W20260812 19:59:33.858966 30743 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:33.859349 30743 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:33.859478 30743 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:33.868214 30743 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:33.868322 30743 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:33.868418 30743 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:33.868503 30743 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:33.879215 30743 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:33.881117 30743 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:33.883249 30743 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:33.892493 30751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:33.892565 30752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:33.893698 30754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:33.893909 30743 server_base.cc:1061] running on GCE node
I20260812 19:59:33.894829 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:33.896555 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:33.897832 30743 hybrid_clock.cc:648] HybridClock initialized: now 1786564773897693 us; error 101 us; skew 500 ppm
I20260812 19:59:33.898208 30743 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:33.900579 30743 webserver.cc:533] Webserver started at http://127.25.14.61:42879/ using document root <none> and password file <none>
I20260812 19:59:33.901404 30743 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:33.901548 30743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:33.901968 30743 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:33.904546 30743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "c81e48d4229949bbb8ebaa5863fe3015"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.905376 30743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "c81e48d4229949bbb8ebaa5863fe3015"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.911432 30743 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 19:59:33.915429 30760 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:33.917214 30743 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:33.917429 30743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "c81e48d4229949bbb8ebaa5863fe3015"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:33.917714 30743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:33.936152 30743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:33.937153 30743 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:33.937479 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:33.959998 30743 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:43755
I20260812 19:59:33.960016 30815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:43755 every 8 connection(s)
I20260812 19:59:33.961802 30743 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:59:33.967164 30816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:33.969400 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30743
I20260812 19:59:33.969607 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:59:33.970768 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44975
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975 with env {}
I20260812 19:59:33.974061 30816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:33.989383 30816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:34.001053 30816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:34.002521 30763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
W20260812 19:59:34.004521 30816 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.054003 30816 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 1
W20260812 19:59:34.056275 30816 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.124594 30816 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 2
W20260812 19:59:34.127017 30816 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.150411 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } attempt: 5
I20260812 19:59:34.157476 30739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:34.159632 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.210932 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 1
W20260812 19:59:34.213428 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.218334 30816 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 3
W20260812 19:59:34.220547 30816 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.281716 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 2
W20260812 19:59:34.284325 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
I20260812 19:59:34.363833 30816 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 4
W20260812 19:59:34.366304 30816 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
W20260812 19:59:34.367071 30819 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:34.367422 30819 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:34.367492 30819 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:34.376560 30819 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:34.376670 30819 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:34.376729 30819 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:34.376768 30819 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:34.387629 30819 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:44975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:34.389359 30819 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:34.390606 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 3
I20260812 19:59:34.391474 30819 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:34.392964 30739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:44975: Network error: Client connection negotiation failed: client connection to 127.25.14.60:44975: connect: Connection refused (error 111)
W20260812 19:59:34.400203 30826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:34.400203 30827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:34.401252 30829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:34.401608 30819 server_base.cc:1061] running on GCE node
I20260812 19:59:34.402597 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:34.404563 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:34.406015 30819 hybrid_clock.cc:648] HybridClock initialized: now 1786564774405851 us; error 118 us; skew 500 ppm
I20260812 19:59:34.406497 30819 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:34.409184 30819 webserver.cc:533] Webserver started at http://127.25.14.60:43107/ using document root <none> and password file <none>
I20260812 19:59:34.410192 30819 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:34.410362 30819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:34.410797 30819 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:34.413666 30819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "bd1b9196dd6e4e8c871f246b5d47bd67"
format_stamp: "Formatted at 2026-08-12 19:59:34 on dist-test-slave-p3t5"
I20260812 19:59:34.414742 30819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "bd1b9196dd6e4e8c871f246b5d47bd67"
format_stamp: "Formatted at 2026-08-12 19:59:34 on dist-test-slave-p3t5"
I20260812 19:59:34.421886 30819 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 19:59:34.426324 30836 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:34.428009 30819 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:59:34.428265 30819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "bd1b9196dd6e4e8c871f246b5d47bd67"
format_stamp: "Formatted at 2026-08-12 19:59:34 on dist-test-slave-p3t5"
I20260812 19:59:34.428557 30819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:34.458904 30819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:34.460129 30819 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:34.460469 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:34.486582 30819 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:44975
I20260812 19:59:34.486590 30894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:44975 every 8 connection(s)
I20260812 19:59:34.488417 30819 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:59:34.493245 30895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:34.496198 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30819
I20260812 19:59:34.496454 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:59:34.501600 30895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:34.516722 30895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:34.524025 30895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:34.535229 30895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: Bootstrap starting.
I20260812 19:59:34.538725 30895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:34.540104 30895 log.cc:826] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:34.543151 30895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: No bootstrap required, opened a new log
I20260812 19:59:34.548759 30895 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.549271 30895 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:34.549432 30895 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1b9196dd6e4e8c871f246b5d47bd67, State: Initialized, Role: FOLLOWER
I20260812 19:59:34.550220 30895 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.551147 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.551455 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.552281 30895 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:34.564311 30914 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:34.564550 30914 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:34.570191 30739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 4
I20260812 19:59:34.563897 30819 master_runner.cc:187] Error getting master registration for 127.25.14.62:38065: OK, instance_id { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" instance_seqno: 1786564773446701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:34.572597 30819 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:34.583021 30739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Bootstrap starting.
I20260812 19:59:34.588270 30739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:34.589884 30739 log.cc:826] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:34.593364 30739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: No bootstrap required, opened a new log
I20260812 19:59:34.601223 30739 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.601552 30739 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:34.601742 30739 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7036daa927394cc2ae6c89c11056be04, State: Initialized, Role: FOLLOWER
I20260812 19:59:34.602450 30739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.603613 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.603960 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.605021 30739 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:34.618999 30931 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:34.619277 30931 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:34.616271 30667 master_runner.cc:187] Error getting master registration for 127.25.14.61:43755: OK, instance_id { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" instance_seqno: 1786564773953836 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:34.623633 30667 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:34.665572 30816 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } attempt: 5
I20260812 19:59:34.680819 30816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: Bootstrap starting.
W20260812 19:59:34.683344 30689 tablet.cc:2406] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:34.685741 30816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:34.687417 30816 log.cc:826] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:34.692386 30816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: No bootstrap required, opened a new log
I20260812 19:59:34.700778 30816 raft_consensus.cc:359] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.701215 30816 raft_consensus.cc:385] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:34.701371 30816 raft_consensus.cc:740] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81e48d4229949bbb8ebaa5863fe3015, State: Initialized, Role: FOLLOWER
I20260812 19:59:34.702527 30816 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.702970 30934 raft_consensus.cc:493] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:59:34.703320 30934 raft_consensus.cc:515] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.706534 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c81e48d4229949bbb8ebaa5863fe3015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7036daa927394cc2ae6c89c11056be04" is_pre_election: true
I20260812 19:59:34.707188 30711 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81e48d4229949bbb8ebaa5863fe3015 in term 0.
I20260812 19:59:34.709900 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.710238 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.711833 30764 leader_election.cc:304] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7036daa927394cc2ae6c89c11056be04, c81e48d4229949bbb8ebaa5863fe3015; no voters:
I20260812 19:59:34.712335 30816 sys_catalog.cc:565] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:34.713229 30842 tablet.cc:2406] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:34.714303 30866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c81e48d4229949bbb8ebaa5863fe3015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" is_pre_election: true
I20260812 19:59:34.714891 30866 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81e48d4229949bbb8ebaa5863fe3015 in term 0.
I20260812 19:59:34.715380 30935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:59:34.715517 30935 raft_consensus.cc:493] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:59:34.715634 30935 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:34.715777 30934 leader_election.cc:290] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7036daa927394cc2ae6c89c11056be04 (127.25.14.62:38065), bd1b9196dd6e4e8c871f246b5d47bd67 (127.25.14.60:44975)
I20260812 19:59:34.719197 30935 raft_consensus.cc:515] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.720635 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c81e48d4229949bbb8ebaa5863fe3015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7036daa927394cc2ae6c89c11056be04"
I20260812 19:59:34.720902 30711 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:34.724979 30711 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81e48d4229949bbb8ebaa5863fe3015 in term 1.
I20260812 19:59:34.725716 30764 leader_election.cc:304] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7036daa927394cc2ae6c89c11056be04, c81e48d4229949bbb8ebaa5863fe3015; no voters:
I20260812 19:59:34.726173 30934 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:59:34.730011 30866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c81e48d4229949bbb8ebaa5863fe3015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67"
I20260812 19:59:34.730280 30866 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:34.735901 30866 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81e48d4229949bbb8ebaa5863fe3015 in term 1.
I20260812 19:59:34.738958 30935 leader_election.cc:290] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [CANDIDATE]: Term 1 election: Requested vote from peers 7036daa927394cc2ae6c89c11056be04 (127.25.14.62:38065), bd1b9196dd6e4e8c871f246b5d47bd67 (127.25.14.60:44975)
I20260812 19:59:34.743557 30934 raft_consensus.cc:697] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 1 LEADER]: Becoming Leader. State: Replica: c81e48d4229949bbb8ebaa5863fe3015, State: Running, Role: LEADER
I20260812 19:59:34.744230 30934 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } }
I20260812 19:59:34.755127 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81e48d4229949bbb8ebaa5863fe3015. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.755493 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:34.766621 30942 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:34.791780 30942 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:34.817968 30866 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:34.819416 30935 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:34.834097 30711 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:34.838660 30935 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:34.850723 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81e48d4229949bbb8ebaa5863fe3015. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.851027 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.866918 30939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.867302 30939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:34.874449 30942 catalog_manager.cc:1383] Generated new cluster ID: f26eefcecc5448ccb805e11c4bb7ddc4
I20260812 19:59:34.874599 30942 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:34.878191 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.878466 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.887070 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81e48d4229949bbb8ebaa5863fe3015. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.887460 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.898962 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.899257 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:34.910235 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } }
I20260812 19:59:34.910519 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:34.920302 30942 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:59:34.922217 30942 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:34.945519 30942 catalog_manager.cc:6092] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: Generated new TSK 0
I20260812 19:59:34.946784 30942 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:59:34.961565 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 19:59:35.521975 30961 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:35.522296 30961 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:35.522447 30961 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:35.531402 30961 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:35.531623 30961 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:35.542395 30961 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:35.544212 30961 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:35.546844 30961 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:35.557590 30968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:35.558025 30970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:35.558461 30967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:35.559283 30961 server_base.cc:1061] running on GCE node
I20260812 19:59:35.560005 30961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:35.561234 30961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:35.562456 30961 hybrid_clock.cc:648] HybridClock initialized: now 1786564775562297 us; error 122 us; skew 500 ppm
I20260812 19:59:35.562808 30961 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:35.565413 30961 webserver.cc:533] Webserver started at http://127.25.14.1:36149/ using document root <none> and password file <none>
I20260812 19:59:35.566653 30961 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:35.566879 30961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:35.567219 30914 catalog_manager.cc:1295] Loaded cluster ID: f26eefcecc5448ccb805e11c4bb7ddc4
I20260812 19:59:35.567332 30914 catalog_manager.cc:1588] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: loading cluster ID for follower catalog manager: success
I20260812 19:59:35.567373 30961 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:35.570947 30961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:35.571357 30914 catalog_manager.cc:1610] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: acquiring CA information for follower catalog manager: success
I20260812 19:59:35.572078 30961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:35.574020 30914 catalog_manager.cc:1638] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:35.579006 30961 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 19:59:35.582988 30977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:35.585213 30961 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:35.585417 30961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:35.585733 30961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:35.612924 30961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:35.614167 30961 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:35.614530 30961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:35.615705 30961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:35.617980 30961 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:35.618108 30961 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:35.618260 30961 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:35.618386 30961 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:35.621366 30931 catalog_manager.cc:1295] Loaded cluster ID: f26eefcecc5448ccb805e11c4bb7ddc4
I20260812 19:59:35.621505 30931 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: loading cluster ID for follower catalog manager: success
I20260812 19:59:35.625290 30931 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: acquiring CA information for follower catalog manager: success
I20260812 19:59:35.628082 30931 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:35.658609 30961 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39823
I20260812 19:59:35.658668 31099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39823 every 8 connection(s)
I20260812 19:59:35.660303 30961 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:35.673692 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 30961
I20260812 19:59:35.674104 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:59:35.683905 31101 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38065
I20260812 19:59:35.684495 31101 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:35.685891 31100 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44975
I20260812 19:59:35.686075 31100 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:35.686276 31101 heartbeater.cc:507] Master 127.25.14.62:38065 requested a full tablet report, sending...
I20260812 19:59:35.686683 31100 heartbeater.cc:507] Master 127.25.14.60:44975 requested a full tablet report, sending...
I20260812 19:59:35.687656 31102 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43755
I20260812 19:59:35.687839 31102 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:35.688323 31102 heartbeater.cc:507] Master 127.25.14.61:43755 requested a full tablet report, sending...
I20260812 19:59:35.689214 30701 ts_manager.cc:194] Registered new tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:35.690266 30777 ts_manager.cc:194] Registered new tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:35.690411 30856 ts_manager.cc:194] Registered new tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:35.692394 30777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41307
I20260812 19:59:35.710278 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:35.712620 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:35381
--webserver_interface=127.25.14.59
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381 with env {}
W20260812 19:59:36.109581 31110 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:36.109989 31110 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:36.110069 31110 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:36.119057 31110 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:36.119179 31110 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:36.119235 31110 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:36.119275 31110 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:36.130774 31110 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:36.132615 31110 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:36.134718 31110 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:36.143931 31116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:36.143965 31115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:36.144721 31120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:36.144850 31110 server_base.cc:1061] running on GCE node
I20260812 19:59:36.145852 31110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:36.147647 31110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:36.148912 31110 hybrid_clock.cc:648] HybridClock initialized: now 1786564776148763 us; error 115 us; skew 500 ppm
I20260812 19:59:36.149309 31110 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:36.151935 31110 webserver.cc:533] Webserver started at http://127.25.14.59:35991/ using document root <none> and password file <none>
I20260812 19:59:36.152774 31110 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:36.152940 31110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:36.153347 31110 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:36.156051 31110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
format_stamp: "Formatted at 2026-08-12 19:59:36 on dist-test-slave-p3t5"
I20260812 19:59:36.156960 31110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
format_stamp: "Formatted at 2026-08-12 19:59:36 on dist-test-slave-p3t5"
I20260812 19:59:36.162930 31110 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 19:59:36.167024 31126 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:36.168677 31110 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:36.168972 31110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
format_stamp: "Formatted at 2026-08-12 19:59:36 on dist-test-slave-p3t5"
I20260812 19:59:36.169247 31110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:36.187991 31110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:36.189003 31110 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:36.189320 31110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:36.212899 31110 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:35381
I20260812 19:59:36.212919 31181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:35381 every 8 connection(s)
I20260812 19:59:36.214812 31110 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:36.220072 31182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:36.223740 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31110
I20260812 19:59:36.228507 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:36.240469 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:36.247886 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:36.254709 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:36.265096 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Bootstrap starting.
I20260812 19:59:36.268611 31182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:36.269948 31182 log.cc:826] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:36.272746 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: No bootstrap required, opened a new log
I20260812 19:59:36.278543 31182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } }
I20260812 19:59:36.278941 31182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:36.279069 31182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aa81b7a4cc417cb41e6d4d788d9537, State: Initialized, Role: FOLLOWER
I20260812 19:59:36.279772 31182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } }
I20260812 19:59:36.280758 31192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } } }
I20260812 19:59:36.281049 31192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:36.281993 31182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:36.293519 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30961
W20260812 19:59:36.293478 31204 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:36.293648 31204 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:36.302466 31110 master_runner.cc:429] Detected that this master 73aa81b7a4cc417cb41e6d4d788d9537 is joining an existing cluster
I20260812 19:59:36.302599 31110 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 19:59:36.303747 31110 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 19:59:36.304127 31110 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 19:59:36.304338 31095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:36.304348 31110 tablet_replica.cc:333] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: stopping tablet replica
I20260812 19:59:36.306450 31110 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:35381
I20260812 19:59:36.353569 30777 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:35381: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 35381
}
attrs {
promote: true
}
}
dest_uuid: "c81e48d4229949bbb8ebaa5863fe3015"
cas_config_opid_index: -1
I20260812 19:59:36.355239 30777 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } }
I20260812 19:59:36.359344 30711 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 19:59:36.360117 30762 consensus_peers.cc:597] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 -> Peer 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59:35381): Couldn't send request to peer 73aa81b7a4cc417cb41e6d4d788d9537. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:59:36.360296 30935 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:59:36.362924 30866 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 19:59:36.363866 30935 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:59:36.379451 31220 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.384884 30711 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.394603 31220 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:35381
I20260812 19:59:36.394418 30866 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.395532 31110 master.cc:562] Master@127.25.14.59:35381 shutting down...
I20260812 19:59:36.396577 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.396870 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:36.399883 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.400147 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:36.400480 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.400676 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:36.406666 31223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.407086 31223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:36.412943 31110 master.cc:584] Master@127.25.14.59:35381 shutdown complete.
I20260812 19:59:36.413093 31110 master_runner.cc:306] Clearing existing system tablet
I20260812 19:59:36.413472 31110 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:36.413646 31110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:36.416464 31110 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260812 19:59:36.422221 31225 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.422632 31225 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:36.430192 31227 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:36.430619 31110 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:36.430967 31110 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.009s sys 0.003s
I20260812 19:59:36.431100 31110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
format_stamp: "Formatted at 2026-08-12 19:59:36 on dist-test-slave-p3t5"
I20260812 19:59:36.434738 31110 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 19:59:36.439651 31110 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 19:59:36.439841 31110 log.cc:1199] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 19:59:36.440280 31110 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Deleting consensus metadata
I20260812 19:59:36.440541 31110 master_runner.cc:316] Copying system tablet from 127.25.14.61:43755
I20260812 19:59:36.443373 31110 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.61:43755
I20260812 19:59:36.452258 30800 tablet_copy_service.cc:140] P c81e48d4229949bbb8ebaa5863fe3015: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 73aa81b7a4cc417cb41e6d4d788d9537 ({username='slave'} at 127.0.0.1:41822)
I20260812 19:59:36.452627 30800 tablet_copy_service.cc:161] P c81e48d4229949bbb8ebaa5863fe3015: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 73aa81b7a4cc417cb41e6d4d788d9537 at {username='slave'} at 127.0.0.1:41822: session id = 73aa81b7a4cc417cb41e6d4d788d9537-00000000000000000000000000000000
I20260812 19:59:36.456871 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30961
I20260812 19:59:36.468586 30800 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:59:36.471441 31110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 19:59:36.479159 31110 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: tablet copy: Starting download of 0 data blocks...
I20260812 19:59:36.479534 31110 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: tablet copy: Starting download of 1 WAL segments...
I20260812 19:59:36.485500 31110 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:59:36.493254 30800 tablet_copy_service.cc:342] P c81e48d4229949bbb8ebaa5863fe3015: Request end of tablet copy session 73aa81b7a4cc417cb41e6d4d788d9537-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:41822
I20260812 19:59:36.493379 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39823
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--webserver_port=36149
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381 with env {}
I20260812 19:59:36.493942 30800 tablet_copy_service.cc:434] P c81e48d4229949bbb8ebaa5863fe3015: ending tablet copy session 73aa81b7a4cc417cb41e6d4d788d9537-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 73aa81b7a4cc417cb41e6d4d788d9537
W20260812 19:59:36.503077 31110 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 19:59:36.508497 31110 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:36.511894 31235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:36.513015 31236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:36.520107 31238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:36.520298 31110 server_base.cc:1061] running on GCE node
I20260812 19:59:36.520764 31110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:36.521759 31110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:36.522981 31110 hybrid_clock.cc:648] HybridClock initialized: now 1786564776522789 us; error 144 us; skew 500 ppm
I20260812 19:59:36.523234 31110 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:36.524807 31110 webserver.cc:533] Webserver started at http://127.25.14.59:35249/ using document root <none> and password file <none>
I20260812 19:59:36.525214 31110 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:36.525321 31110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:36.528582 31110 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 19:59:36.530798 31247 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:36.531845 31110 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 19:59:36.531989 31110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "73aa81b7a4cc417cb41e6d4d788d9537"
format_stamp: "Formatted at 2026-08-12 19:59:36 on dist-test-slave-p3t5"
I20260812 19:59:36.532166 31110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:36.580147 31110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:36.581025 31110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:36.613984 31110 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:35381
I20260812 19:59:36.614060 31306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:35381 every 8 connection(s)
I20260812 19:59:36.615366 31110 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 19:59:36.621073 31307 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:36.624776 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Bootstrap starting.
I20260812 19:59:36.643079 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:36.643682 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: Bootstrap complete.
I20260812 19:59:36.644893 31307 raft_consensus.cc:359] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } }
I20260812 19:59:36.645169 31307 raft_consensus.cc:740] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73aa81b7a4cc417cb41e6d4d788d9537, State: Initialized, Role: LEARNER
I20260812 19:59:36.645521 31307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } }
I20260812 19:59:36.646556 31310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.646909 31307 sys_catalog.cc:565] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:36.646974 31310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: This master's current role is: LEARNER
I20260812 19:59:36.656682 31322 catalog_manager.cc:1295] Loaded cluster ID: f26eefcecc5448ccb805e11c4bb7ddc4
I20260812 19:59:36.656819 31322 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: loading cluster ID for follower catalog manager: success
I20260812 19:59:36.660853 31322 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: acquiring CA information for follower catalog manager: success
I20260812 19:59:36.663308 31322 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 19:59:36.927124 31241 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:36.927577 31241 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:36.927729 31241 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:36.937515 31241 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:36.937881 31241 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:36.944379 31279 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 19:59:36.945237 31310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81e48d4229949bbb8ebaa5863fe3015. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:36.945477 31310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: This master's current role is: LEARNER
I20260812 19:59:36.950438 31241 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36149
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:36.952381 31241 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:36.954449 31241 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:36.965399 31330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:36.966346 31331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:36.968401 31241 server_base.cc:1061] running on GCE node
W20260812 19:59:36.968770 31333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:36.969683 31241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:36.971393 31241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:36.974824 31241 hybrid_clock.cc:648] HybridClock initialized: now 1786564776974581 us; error 185 us; skew 500 ppm
I20260812 19:59:36.975283 31241 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:36.978993 31241 webserver.cc:533] Webserver started at http://127.25.14.1:36149/ using document root <none> and password file <none>
I20260812 19:59:36.980069 31241 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:36.980219 31241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:36.986732 31241 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:36.989907 31339 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:36.991616 31241 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:36.991793 31241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:36.992565 31241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:37.042822 31241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:37.043751 31241 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:37.044030 31241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:37.045061 31241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:37.047224 31241 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:37.047313 31241 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:37.047406 31241 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:37.047456 31241 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:37.086416 31241 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39823
I20260812 19:59:37.086459 31468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39823 every 8 connection(s)
I20260812 19:59:37.088227 31241 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:37.097400 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31241
I20260812 19:59:37.138514 31470 heartbeater.cc:344] Connected to a master server at 127.25.14.59:35381
I20260812 19:59:37.139024 31470 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:37.140609 31470 heartbeater.cc:507] Master 127.25.14.59:35381 requested a full tablet report, sending...
I20260812 19:59:37.143698 31266 ts_manager.cc:194] Registered new tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:37.145234 31469 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44975
I20260812 19:59:37.145406 31469 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:37.146067 31469 heartbeater.cc:507] Master 127.25.14.60:44975 requested a full tablet report, sending...
I20260812 19:59:37.146852 31471 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38065
I20260812 19:59:37.147019 31471 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:37.147634 31471 heartbeater.cc:507] Master 127.25.14.62:38065 requested a full tablet report, sending...
I20260812 19:59:37.147711 30856 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:37.149242 30701 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:37.151072 31472 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43755
I20260812 19:59:37.151122 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:37.151222 31472 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:37.151641 31472 heartbeater.cc:507] Master 127.25.14.61:43755 requested a full tablet report, sending...
I20260812 19:59:37.152792 30777 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:37.153991 30777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:41201
I20260812 19:59:37.159816 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 19:59:37.160017 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30667
I20260812 19:59:37.160966 30733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:37.273249 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30667
I20260812 19:59:37.305714 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38065
--webserver_interface=127.25.14.62
--webserver_port=46473
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381 with env {}
W20260812 19:59:37.343273 30764 consensus_peers.cc:597] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 -> Peer 7036daa927394cc2ae6c89c11056be04 (127.25.14.62:38065): Couldn't send request to peer 7036daa927394cc2ae6c89c11056be04. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:38065: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:59:37.535441 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:37.536391 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:37.537511 31486 raft_consensus.cc:1064] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015: attempting to promote NON_VOTER 73aa81b7a4cc417cb41e6d4d788d9537 to VOTER
I20260812 19:59:37.538539 31486 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } }
I20260812 19:59:37.541689 30866 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:37.541910 31279 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 1 LEARNER]: Refusing update from remote peer c81e48d4229949bbb8ebaa5863fe3015: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 19:59:37.542641 31490 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 19:59:37.543139 30764 consensus_peers.cc:597] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 -> Peer 7036daa927394cc2ae6c89c11056be04 (127.25.14.62:38065): Couldn't send request to peer 7036daa927394cc2ae6c89c11056be04. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:38065: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:59:37.543164 31486 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:59:37.547695 31486 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.548758 31279 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.552012 31222 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.552299 31222 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:37.552754 31222 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.552960 31222 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:37.553339 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.553519 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:37.553141 30866 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.559145 31495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.559469 31495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa81b7a4cc417cb41e6d4d788d9537 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:37.560045 31494 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:37.560351 31494 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b9196dd6e4e8c871f246b5d47bd67 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:37.727641 31485 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:37.728056 31485 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:37.728137 31485 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:37.738415 31485 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:37.738529 31485 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:37.738593 31485 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:37.738638 31485 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:37.750869 31485 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=46473
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:37.752823 31485 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:37.755062 31485 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:37.765033 31501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:37.765117 31500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:37.766214 31503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:37.767657 31485 server_base.cc:1061] running on GCE node
I20260812 19:59:37.768739 31485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:37.770843 31485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:37.772250 31485 hybrid_clock.cc:648] HybridClock initialized: now 1786564777772092 us; error 118 us; skew 500 ppm
I20260812 19:59:37.772686 31485 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:37.775360 31485 webserver.cc:533] Webserver started at http://127.25.14.62:46473/ using document root <none> and password file <none>
I20260812 19:59:37.776377 31485 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:37.776559 31485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:37.783059 31485 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 19:59:37.786682 31509 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:37.788473 31485 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 19:59:37.788726 31485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "7036daa927394cc2ae6c89c11056be04"
format_stamp: "Formatted at 2026-08-12 19:59:33 on dist-test-slave-p3t5"
I20260812 19:59:37.789553 31485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:37.829407 31485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:37.830508 31485 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:37.830868 31485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:37.854882 31485 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38065
I20260812 19:59:37.854930 31564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38065 every 8 connection(s)
I20260812 19:59:37.856884 31485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:37.858366 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31485
I20260812 19:59:37.858718 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31241
I20260812 19:59:37.867230 31565 sys_catalog.cc:263] Verifying existing consensus state
I20260812 19:59:37.869340 31464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:37.872304 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Bootstrap starting.
I20260812 19:59:37.899433 31565 log.cc:826] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:37.912448 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:59:37.913249 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: Bootstrap complete.
I20260812 19:59:37.924171 31565 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } }
I20260812 19:59:37.925267 31565 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7036daa927394cc2ae6c89c11056be04, State: Initialized, Role: FOLLOWER
I20260812 19:59:37.926333 31565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } }
I20260812 19:59:37.927367 31573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: true } } }
I20260812 19:59:37.927722 31573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:37.928889 31565 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:37.947350 31586 catalog_manager.cc:1295] Loaded cluster ID: f26eefcecc5448ccb805e11c4bb7ddc4
I20260812 19:59:37.947502 31586 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: loading cluster ID for follower catalog manager: success
I20260812 19:59:37.952481 31586 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: acquiring CA information for follower catalog manager: success
I20260812 19:59:37.955442 31586 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 19:59:37.999080 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31241
I20260812 19:59:38.031176 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39823
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--webserver_port=36149
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381 with env {}
I20260812 19:59:38.164578 31539 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73aa81b7a4cc417cb41e6d4d788d9537 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:38.172998 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:38.173413 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:38.174031 31573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81e48d4229949bbb8ebaa5863fe3015. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:38.174257 31573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036daa927394cc2ae6c89c11056be04 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 19:59:38.434474 31595 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:38.434866 31595 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:38.435032 31595 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:38.444023 31595 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:38.444257 31595 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:38.455435 31595 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36149
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:38.457237 31595 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:38.459327 31595 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:38.469985 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:38.469992 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:38.471135 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:38.471870 31595 server_base.cc:1061] running on GCE node
I20260812 19:59:38.472627 31595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:38.473963 31595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:38.475209 31595 hybrid_clock.cc:648] HybridClock initialized: now 1786564778475068 us; error 105 us; skew 500 ppm
I20260812 19:59:38.475575 31595 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:38.478088 31595 webserver.cc:533] Webserver started at http://127.25.14.1:36149/ using document root <none> and password file <none>
I20260812 19:59:38.478936 31595 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:38.479084 31595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:38.485391 31595 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 19:59:38.489497 31613 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:38.491554 31595 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:59:38.491778 31595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:38.492874 31595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:38.520596 31595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:38.521541 31595 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:38.521898 31595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:38.523020 31595 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:38.525053 31595 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:38.525128 31595 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:38.525218 31595 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:38.525264 31595 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:38.565573 31595 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39823
I20260812 19:59:38.565605 31739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39823 every 8 connection(s)
I20260812 19:59:38.567972 31595 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:38.567978 31490 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81e48d4229949bbb8ebaa5863fe3015" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7036daa927394cc2ae6c89c11056be04" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38065 } } peers { permanent_uuid: "c81e48d4229949bbb8ebaa5863fe3015" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43755 } } peers { permanent_uuid: "bd1b9196dd6e4e8c871f246b5d47bd67" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 44975 } } peers { permanent_uuid: "73aa81b7a4cc417cb41e6d4d788d9537" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 35381 } attrs { promote: false } } }
I20260812 19:59:38.568303 31490 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81e48d4229949bbb8ebaa5863fe3015 [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:38.575011 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31595
I20260812 19:59:38.575407 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31595
I20260812 19:59:38.622370 31743 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44975
I20260812 19:59:38.623036 31743 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:38.624609 31743 heartbeater.cc:507] Master 127.25.14.60:44975 requested a full tablet report, sending...
I20260812 19:59:38.627465 30856 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:38.629458 31744 heartbeater.cc:344] Connected to a master server at 127.25.14.59:35381
I20260812 19:59:38.629652 31744 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:38.630309 31744 heartbeater.cc:507] Master 127.25.14.59:35381 requested a full tablet report, sending...
I20260812 19:59:38.632041 31266 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:38.637629 31735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:38.637764 31745 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38065
I20260812 19:59:38.728318 31745 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:38.729056 31745 heartbeater.cc:507] Master 127.25.14.62:38065 requested a full tablet report, sending...
I20260812 19:59:38.730976 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31595
I20260812 19:59:38.731727 31528 ts_manager.cc:194] Registered new tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
W20260812 19:59:38.756273 31511 connection.cc:573] server connection from 127.25.14.1:58285 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:38.757161 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39823
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--webserver_port=36149
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--fault_before_start=1.0 with env {}
W20260812 19:59:38.757845 31756 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 19:59:38.637669 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:47243 (local address 127.25.14.61:43755)
0812 19:59:38.638664 (+ 995us) server_negotiation.cc:207] Beginning negotiation
0812 19:59:38.638672 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 19:59:38.638696 (+ 24us) server_negotiation.cc:408] Connection header received
0812 19:59:38.638766 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 19:59:38.638776 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 19:59:38.638875 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 19:59:38.639355 (+ 480us) server_negotiation.cc:227] Negotiated authn=SASL
0812 19:59:38.729337 (+ 89982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 19:59:38.738414 (+ 9077us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 19:59:38.756746 (+ 18332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:47243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":716,"threads_started":1}
W20260812 19:59:39.152036 31757 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:39.152437 31757 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:39.152580 31757 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 19:59:39.152676 31757 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:39.161593 31757 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:39.161953 31757 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:39.172592 31757 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36149
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:39.174489 31757 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:39.176429 31757 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:39.187179 31763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:39.187391 31765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:39.187203 31762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:39.188911 31757 server_base.cc:1061] running on GCE node
I20260812 19:59:39.189558 31757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:39.191804 31757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:39.193198 31757 hybrid_clock.cc:648] HybridClock initialized: now 1786564779193020 us; error 124 us; skew 500 ppm
I20260812 19:59:39.193571 31757 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:39.196069 31757 webserver.cc:533] Webserver started at http://127.25.14.1:36149/ using document root <none> and password file <none>
I20260812 19:59:39.196902 31757 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:39.197053 31757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:39.203151 31757 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 19:59:39.206349 31771 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:39.208037 31757 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:39.208350 31757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:39.209201 31757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:39.232285 31757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:39.233498 31757 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:39.233852 31757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:39.234964 31757 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:39.236830 31757 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:39.236936 31757 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:39.237097 31757 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:39.237193 31757 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 19:59:39.237594 31757 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 19:59:39.237735 31757 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 19:59:39.237826 31757 fault_injection.cc:59] injecting fault for kudu (pid 31757): FLAGS_fault_before_start (process will exit)
I20260812 19:59:39.261821 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:39823
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975
--webserver_port=36149
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381 with env {}
W20260812 19:59:39.656157 31779 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:39.656545 31779 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:39.656697 31779 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:39.665417 31779 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:39.665688 31779 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:39.676219 31779 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35161,127.25.14.21:46401,127.25.14.22:33497,127.25.14.23:46789,127.25.14.24:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:39823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36149
--tserver_master_addrs=127.25.14.62:38065,127.25.14.61:43755,127.25.14.60:44975,127.25.14.59:35381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:39.678166 31779 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:39.680269 31779 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:39.691984 31787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:39.692133 31786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:39.695194 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:39.696178 31779 server_base.cc:1061] running on GCE node
I20260812 19:59:39.696877 31779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 19:59:39.698236 31779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 19:59:39.699553 31779 hybrid_clock.cc:648] HybridClock initialized: now 1786564779699398 us; error 119 us; skew 500 ppm
I20260812 19:59:39.699966 31779 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:39.702527 31779 webserver.cc:533] Webserver started at http://127.25.14.1:36149/ using document root <none> and password file <none>
I20260812 19:59:39.703307 31779 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:39.703413 31779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:39.709836 31779 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 19:59:39.712862 31796 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:39.714529 31779 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:39.714684 31779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "5c6945ad1bd74d75b4685c918e783e1d"
format_stamp: "Formatted at 2026-08-12 19:59:35 on dist-test-slave-p3t5"
I20260812 19:59:39.715507 31779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:39.732123 31779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:39.733295 31779 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:39.733573 31779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:39.734786 31779 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:39.737001 31779 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:39.737087 31779 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:39.737180 31779 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:39.737226 31779 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:39.775590 31779 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39823
I20260812 19:59:39.775688 31923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39823 every 8 connection(s)
I20260812 19:59:39.777297 31779 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:39.783977 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 31779
I20260812 19:59:39.784390 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31779
I20260812 19:59:39.814831 31926 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38065
I20260812 19:59:39.815397 31926 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:39.816959 31926 heartbeater.cc:507] Master 127.25.14.62:38065 requested a full tablet report, sending...
I20260812 19:59:39.817803 31924 heartbeater.cc:344] Connected to a master server at 127.25.14.60:44975
I20260812 19:59:39.818005 31924 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:39.818725 31924 heartbeater.cc:507] Master 127.25.14.60:44975 requested a full tablet report, sending...
I20260812 19:59:39.819710 31925 heartbeater.cc:344] Connected to a master server at 127.25.14.59:35381
I20260812 19:59:39.820068 31925 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:39.820607 31925 heartbeater.cc:507] Master 127.25.14.59:35381 requested a full tablet report, sending...
I20260812 19:59:39.820987 30856 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:39.821218 31528 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:39.821889 31266 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:39.828027 31927 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43755
I20260812 19:59:39.828251 31927 heartbeater.cc:461] Registering TS with master...
I20260812 19:59:39.828799 31927 heartbeater.cc:507] Master 127.25.14.61:43755 requested a full tablet report, sending...
I20260812 19:59:39.830060 30777 ts_manager.cc:194] Re-registered known tserver with Master: 5c6945ad1bd74d75b4685c918e783e1d (127.25.14.1:39823)
I20260812 19:59:39.830878 31919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:39.831130 30777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:44873
I20260812 19:59:39.921499 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31779
I20260812 19:59:39.946141 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31485
I20260812 19:59:39.952661 31560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:40.063778 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31485
I20260812 19:59:40.093304 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30743
I20260812 19:59:40.094830 30810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:40.208217 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30743
W20260812 19:59:40.237924 30839 connection.cc:573] server connection from 127.0.0.1:58188 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 19:59:40.237924 31248 connection.cc:573] server connection from 127.0.0.1:42742 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 19:59:40.238354 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 30819
I20260812 19:59:40.239547 30889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:40.352330 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 30819
I20260812 19:59:40.380434 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 31110
I20260812 19:59:40.381827 31302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 19:59:40.497085 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 31110
2026-08-12T19:59:40Z chronyd exiting
2026-08-12T19:59:40Z chronyd exiting
2026-08-12T19:59:40Z chronyd exiting
2026-08-12T19:59:40Z chronyd exiting
2026-08-12T19:59:40Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/2 (7937 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 19:59:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 19:59:57.320920 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34197
--webserver_interface=127.25.14.62
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583 with env {}
W20260812 19:59:57.711972 32265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:57.712411 32265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:57.712558 32265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:57.721537 32265 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:57.721698 32265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:57.721814 32265 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:57.721897 32265 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:57.732786 32265 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:57.734763 32265 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:57.736783 32265 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:57.746757 32270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:57.746888 32271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:57.747496 32273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:57.747882 32265 server_base.cc:1061] running on GCE node
I20260812 19:59:57.748770 32265 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:57.748996 32265 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:57.749107 32265 hybrid_clock.cc:648] HybridClock initialized: now 1786564797749083 us; error 0 us; skew 500 ppm
I20260812 19:59:57.749514 32265 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:57.752794 32265 webserver.cc:533] Webserver started at http://127.25.14.62:44557/ using document root <none> and password file <none>
I20260812 19:59:57.753810 32265 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:57.753957 32265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:57.754359 32265 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:57.756973 32265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "5388a34d587143f9a6b1bdb1663f249f"
format_stamp: "Formatted at 2026-08-12 19:59:57 on dist-test-slave-p3t5"
I20260812 19:59:57.757916 32265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "5388a34d587143f9a6b1bdb1663f249f"
format_stamp: "Formatted at 2026-08-12 19:59:57 on dist-test-slave-p3t5"
I20260812 19:59:57.763993 32265 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:59:57.767809 32280 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:57.769397 32265 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:59:57.769668 32265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5388a34d587143f9a6b1bdb1663f249f"
format_stamp: "Formatted at 2026-08-12 19:59:57 on dist-test-slave-p3t5"
I20260812 19:59:57.769955 32265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:57.789341 32265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:57.790541 32265 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:57.790881 32265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:57.815089 32265 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34197
I20260812 19:59:57.815111 32339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34197 every 8 connection(s)
I20260812 19:59:57.817107 32265 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 19:59:57.822136 32340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:57.823020 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32265
I20260812 19:59:57.823247 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 19:59:57.824314 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43631
--webserver_interface=127.25.14.61
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583 with env {}
I20260812 19:59:57.828714 32340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:57.848348 32340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:57.849732 32283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
W20260812 19:59:57.851711 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43631: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
I20260812 19:59:57.901181 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } attempt: 1
W20260812 19:59:57.903697 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43631: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
I20260812 19:59:57.972048 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } attempt: 2
W20260812 19:59:57.974462 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43631: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
I20260812 19:59:58.065845 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } attempt: 3
W20260812 19:59:58.068307 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43631: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
I20260812 19:59:58.211647 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } attempt: 4
W20260812 19:59:58.213963 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:43631: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111)
W20260812 19:59:58.225474 32345 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:58.225983 32345 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:58.226073 32345 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:58.234815 32345 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:58.234916 32345 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:58.234975 32345 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:58.235016 32345 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:58.245262 32345 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:58.247115 32345 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:58.249047 32345 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:58.257993 32352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:58.258019 32351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:58.258837 32354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:58.259498 32345 server_base.cc:1061] running on GCE node
I20260812 19:59:58.260375 32345 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:58.260583 32345 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:58.260686 32345 hybrid_clock.cc:648] HybridClock initialized: now 1786564798260644 us; error 0 us; skew 500 ppm
I20260812 19:59:58.261086 32345 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:58.264569 32345 webserver.cc:533] Webserver started at http://127.25.14.61:45907/ using document root <none> and password file <none>
I20260812 19:59:58.265435 32345 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:58.265545 32345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:58.265988 32345 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:58.268651 32345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "3f4627e14c65429a851c75fdcb043ff0"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.269520 32345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "3f4627e14c65429a851c75fdcb043ff0"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.275614 32345 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:59:58.279635 32359 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:58.281155 32345 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:59:58.281376 32345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "3f4627e14c65429a851c75fdcb043ff0"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.281661 32345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:58.303952 32345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:58.305022 32345 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:58.305341 32345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:58.328052 32345 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:43631
I20260812 19:59:58.328073 32415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:43631 every 8 connection(s)
I20260812 19:59:58.330049 32345 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 19:59:58.333828 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32345
I20260812 19:59:58.334049 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 19:59:58.335085 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37583
--webserver_interface=127.25.14.60
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583 with env {}
I20260812 19:59:58.335475 32417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:58.342038 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:58.358140 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:58.369199 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:58.370684 32363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
W20260812 19:59:58.372684 32417 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.422089 32417 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 1
W20260812 19:59:58.424474 32417 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.492765 32417 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 2
W20260812 19:59:58.495245 32417 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.513294 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } attempt: 5
I20260812 19:59:58.520880 32340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
W20260812 19:59:58.523138 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.574441 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 1
W20260812 19:59:58.576856 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.586503 32417 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 3
W20260812 19:59:58.589269 32417 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.645154 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 2
W20260812 19:59:58.647846 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
I20260812 19:59:58.732532 32417 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 4
W20260812 19:59:58.735025 32417 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
W20260812 19:59:58.753541 32420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:58.754038 32420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:58.754189 32420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 19:59:58.754097 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 3
W20260812 19:59:58.756444 32340 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37583: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37583: connect: Connection refused (error 111)
W20260812 19:59:58.764598 32420 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 19:59:58.764756 32420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:58.764863 32420 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 19:59:58.764948 32420 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 19:59:58.775769 32420 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:58.777779 32420 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:58.780007 32420 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:58.791162 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:58.791147 32429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:58.791430 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:58.792608 32420 server_base.cc:1061] running on GCE node
I20260812 19:59:58.793569 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:58.793838 32420 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:58.793962 32420 hybrid_clock.cc:648] HybridClock initialized: now 1786564798793933 us; error 0 us; skew 500 ppm
I20260812 19:59:58.794330 32420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:58.797366 32420 webserver.cc:533] Webserver started at http://127.25.14.60:37333/ using document root <none> and password file <none>
I20260812 19:59:58.798352 32420 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:58.798468 32420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:58.798794 32420 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:58.801330 32420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.802237 32420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.808286 32420 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 19:59:58.812048 32436 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:58.813690 32420 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:59:58.813917 32420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc"
format_stamp: "Formatted at 2026-08-12 19:59:58 on dist-test-slave-p3t5"
I20260812 19:59:58.814191 32420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:58.840142 32420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:58.841202 32420 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:58.841460 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:58.863868 32420 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:37583
I20260812 19:59:58.863917 32496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:37583 every 8 connection(s)
I20260812 19:59:58.865725 32420 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 19:59:58.870487 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:59:58.872110 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32420
I20260812 19:59:58.872340 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 19:59:58.878825 32497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:58.894006 32497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:58.901759 32497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260812 19:59:58.913540 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Bootstrap starting.
I20260812 19:59:58.917119 32497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:58.918418 32497 log.cc:826] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:58.922099 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: No bootstrap required, opened a new log
I20260812 19:59:58.927798 32497 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:58.928208 32497 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:58.928373 32497 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dadc22df3bb4263b425bf5d8ef0d1bc, State: Initialized, Role: FOLLOWER
I20260812 19:59:58.929499 32497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:58.930668 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:58.930989 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:58.931846 32497 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:59:58.933730 32340 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 4
W20260812 19:59:58.946216 32515 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:58.946376 32515 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:58.948596 32340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Bootstrap starting.
I20260812 19:59:58.943576 32420 master_runner.cc:187] Error getting master registration for 127.25.14.62:34197: OK, instance_id { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" instance_seqno: 1786564797808274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:58.950800 32420 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:58.953346 32340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:58.954895 32340 log.cc:826] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:58.958166 32340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: No bootstrap required, opened a new log
I20260812 19:59:58.966452 32340 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:58.966892 32340 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:58.967065 32340 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388a34d587143f9a6b1bdb1663f249f, State: Initialized, Role: FOLLOWER
I20260812 19:59:58.967867 32340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:58.968892 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:58.969162 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:58.970464 32340 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:58.983541 32530 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:58.983807 32530 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:58.982997 32265 master_runner.cc:187] Error getting master registration for 127.25.14.61:43631: OK, instance_id { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" instance_seqno: 1786564798321454 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 19:59:58.990877 32265 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 19:59:59.034262 32417 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } attempt: 5
W20260812 19:59:59.044387 32286 tablet.cc:2406] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:59.048537 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: Bootstrap starting.
I20260812 19:59:59.053162 32417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: Neither blocks nor log segments found. Creating new log.
I20260812 19:59:59.054847 32417 log.cc:826] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: Log is configured to *not* fsync() on all Append() calls
I20260812 19:59:59.061096 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: No bootstrap required, opened a new log
I20260812 19:59:59.069923 32417 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:59.070358 32417 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:59:59.070508 32417 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4627e14c65429a851c75fdcb043ff0, State: Initialized, Role: FOLLOWER
I20260812 19:59:59.071455 32417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:59.072710 32533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.073155 32533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:59.074218 32417 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 19:59:59.093961 32443 tablet.cc:2406] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:59:59.101022 32345 master_runner.cc:264] No leader master found from master 3f4627e14c65429a851c75fdcb043ff0
I20260812 19:59:59.101181 32345 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 19:59:59.104297 32545 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 19:59:59.104568 32545 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 19:59:59.138857 32504 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:59:59.139209 32504 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:59.141278 32504 leader_election.cc:290] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388a34d587143f9a6b1bdb1663f249f (127.25.14.62:34197), 3f4627e14c65429a851c75fdcb043ff0 (127.25.14.61:43631)
I20260812 19:59:59.142475 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388a34d587143f9a6b1bdb1663f249f" is_pre_election: true
I20260812 19:59:59.143010 32310 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dadc22df3bb4263b425bf5d8ef0d1bc in term 0.
I20260812 19:59:59.143712 32438 leader_election.cc:304] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388a34d587143f9a6b1bdb1663f249f, 6dadc22df3bb4263b425bf5d8ef0d1bc; no voters:
I20260812 19:59:59.145043 32389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4627e14c65429a851c75fdcb043ff0" is_pre_election: true
I20260812 19:59:59.145601 32389 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dadc22df3bb4263b425bf5d8ef0d1bc in term 0.
I20260812 19:59:59.146557 32504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:59:59.146744 32504 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:59:59.146860 32504 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:59.150592 32504 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:59.151968 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388a34d587143f9a6b1bdb1663f249f"
I20260812 19:59:59.152208 32310 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:59.156082 32310 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dadc22df3bb4263b425bf5d8ef0d1bc in term 1.
I20260812 19:59:59.156737 32438 leader_election.cc:304] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388a34d587143f9a6b1bdb1663f249f, 6dadc22df3bb4263b425bf5d8ef0d1bc; no voters:
I20260812 19:59:59.161145 32389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4627e14c65429a851c75fdcb043ff0"
I20260812 19:59:59.161420 32389 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:59:59.165371 32389 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dadc22df3bb4263b425bf5d8ef0d1bc in term 1.
I20260812 19:59:59.167830 32504 leader_election.cc:290] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [CANDIDATE]: Term 1 election: Requested vote from peers 5388a34d587143f9a6b1bdb1663f249f (127.25.14.62:34197), 3f4627e14c65429a851c75fdcb043ff0 (127.25.14.61:43631)
I20260812 19:59:59.168161 32504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:59:59.168352 32504 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 1 LEADER]: Becoming Leader. State: Replica: 6dadc22df3bb4263b425bf5d8ef0d1bc, State: Running, Role: LEADER
I20260812 19:59:59.168972 32504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } }
I20260812 19:59:59.182583 32547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dadc22df3bb4263b425bf5d8ef0d1bc. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.182997 32547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:59.190299 32549 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 19:59:59.195551 32549 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 19:59:59.212249 32389 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:59.213495 32547 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:59.221122 32310 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:59:59.223678 32547 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:59:59.253688 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dadc22df3bb4263b425bf5d8ef0d1bc. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.254006 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:59.258906 32552 mvcc.cc:204] Tried to move back new op lower bound from 7317769417558499328 to 7317769417412501504. Current Snapshot: MvccSnapshot[applied={T|T < 7317769417558499328}]
I20260812 19:59:59.261864 32550 mvcc.cc:204] Tried to move back new op lower bound from 7317769417558499328 to 7317769417412501504. Current Snapshot: MvccSnapshot[applied={T|T < 7317769417558499328}]
I20260812 19:59:59.266109 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.266463 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:59.268796 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.269064 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:59.272076 32533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dadc22df3bb4263b425bf5d8ef0d1bc. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.272284 32549 catalog_manager.cc:1383] Generated new cluster ID: bb0dff71d65a41408b8ca70d7b1faa4d
I20260812 19:59:59.272408 32549 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 19:59:59.272384 32533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:59.288133 32547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.288434 32547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 19:59:59.305815 32549 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 19:59:59.307896 32549 catalog_manager.cc:1540] Loading token signing keys...
I20260812 19:59:59.311551 32533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } }
I20260812 19:59:59.311810 32533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 19:59:59.384613 32549 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Generated new TSK 0
I20260812 19:59:59.386013 32549 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 19:59:59.404872 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--time_source=system_unsync with env {}
W20260812 19:59:59.853044 32560 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 19:59:59.853456 32560 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 19:59:59.853600 32560 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 19:59:59.862434 32560 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 19:59:59.862665 32560 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 19:59:59.873411 32560 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 19:59:59.875294 32560 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:59:59.877384 32560 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 19:59:59.888208 32565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:59:59.888837 32566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:59.889923 32560 server_base.cc:1061] running on GCE node
W20260812 19:59:59.890718 32568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:59:59.891484 32560 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:59:59.891635 32560 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:59:59.891706 32560 hybrid_clock.cc:648] HybridClock initialized: now 1786564799891687 us; error 0 us; skew 500 ppm
I20260812 19:59:59.891986 32560 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:59:59.894531 32560 webserver.cc:533] Webserver started at http://127.25.14.1:34189/ using document root <none> and password file <none>
I20260812 19:59:59.895320 32560 fs_manager.cc:362] Metadata directory not provided
I20260812 19:59:59.895432 32560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:59:59.895771 32560 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:59:59.898402 32560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 19:59:59.899226 32560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 19:59:59.905427 32560 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 19:59:59.909323 32575 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:59.911149 32560 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 19:59:59.911403 32560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 19:59:59.911671 32560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:59:59.938439 32560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 19:59:59.939459 32560 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:59:59.939772 32560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:59:59.940832 32560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 19:59:59.942809 32560 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:59:59.942948 32560 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:59.943097 32560 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 19:59:59.943195 32560 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:59:59.983417 32560 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:44125
I20260812 19:59:59.983479 32701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:44125 every 8 connection(s)
I20260812 19:59:59.985213 32560 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 19:59:59.986042 32530 catalog_manager.cc:1295] Loaded cluster ID: bb0dff71d65a41408b8ca70d7b1faa4d
I20260812 19:59:59.986172 32530 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: loading cluster ID for follower catalog manager: success
I20260812 19:59:59.989678 32530 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: acquiring CA information for follower catalog manager: success
I20260812 19:59:59.989725 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32560
I20260812 19:59:59.989972 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 19:59:59.992775 32530 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:00.017189 32702 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37583
I20260812 20:00:00.017367 32703 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34197
I20260812 20:00:00.017728 32702 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:00.017729 32703 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:00.018007 32707 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43631
I20260812 20:00:00.018199 32707 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:00.019254 32703 heartbeater.cc:507] Master 127.25.14.62:34197 requested a full tablet report, sending...
I20260812 20:00:00.019294 32707 heartbeater.cc:507] Master 127.25.14.61:43631 requested a full tablet report, sending...
I20260812 20:00:00.019320 32702 heartbeater.cc:507] Master 127.25.14.60:37583 requested a full tablet report, sending...
I20260812 20:00:00.021886 32298 ts_manager.cc:194] Registered new tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:00.021898 32379 ts_manager.cc:194] Registered new tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:00.021883 32456 ts_manager.cc:194] Registered new tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:00.023658 32456 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:51275
I20260812 20:00:00.038376 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:00.040791 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:36123
--webserver_interface=127.25.14.59
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123 with env {}
I20260812 20:00:00.106787 32545 catalog_manager.cc:1295] Loaded cluster ID: bb0dff71d65a41408b8ca70d7b1faa4d
I20260812 20:00:00.106940 32545 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: loading cluster ID for follower catalog manager: success
I20260812 20:00:00.109774 32545 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: acquiring CA information for follower catalog manager: success
I20260812 20:00:00.111764 32545 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 20:00:00.441819 32713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:00.442207 32713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:00.442399 32713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:00.451571 32713 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:00.451675 32713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:00.451769 32713 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:00.451875 32713 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:00.462800 32713 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:36123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:00.464681 32713 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:00.466741 32713 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:00.476655 32718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:00.476881 32721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:00.476635 32719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:00.478164 32713 server_base.cc:1061] running on GCE node
I20260812 20:00:00.479132 32713 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:00.479358 32713 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:00.479460 32713 hybrid_clock.cc:648] HybridClock initialized: now 1786564800479437 us; error 0 us; skew 500 ppm
I20260812 20:00:00.479888 32713 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:00.483234 32713 webserver.cc:533] Webserver started at http://127.25.14.59:38977/ using document root <none> and password file <none>
I20260812 20:00:00.484148 32713 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:00.484297 32713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:00.484826 32713 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:00.487905 32713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "20d6a12c6f2b484991fd049c4273e503"
format_stamp: "Formatted at 2026-08-12 20:00:00 on dist-test-slave-p3t5"
I20260812 20:00:00.488840 32713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "20d6a12c6f2b484991fd049c4273e503"
format_stamp: "Formatted at 2026-08-12 20:00:00 on dist-test-slave-p3t5"
I20260812 20:00:00.495375 32713 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 20:00:00.499539 32727 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:00.501292 32713 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:00:00.501485 32713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d6a12c6f2b484991fd049c4273e503"
format_stamp: "Formatted at 2026-08-12 20:00:00 on dist-test-slave-p3t5"
I20260812 20:00:00.501798 32713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:00.531143 32713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:00.532220 32713 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:00.532523 32713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:00.556408 32713 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:36123
I20260812 20:00:00.556432 319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:36123 every 8 connection(s)
I20260812 20:00:00.558753 32713 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:00.563920 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 32713
I20260812 20:00:00.566282 320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:00.575091 320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:00.586510 320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:00.593950 320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:00.601218 320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:00.612213 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Bootstrap starting.
I20260812 20:00:00.615804 320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:00.617084 320 log.cc:826] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:00.620182 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: No bootstrap required, opened a new log
I20260812 20:00:00.626626 320 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } }
I20260812 20:00:00.627017 320 raft_consensus.cc:385] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:00.627111 320 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d6a12c6f2b484991fd049c4273e503, State: Initialized, Role: FOLLOWER
I20260812 20:00:00.627739 320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } }
I20260812 20:00:00.628724 329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } } }
I20260812 20:00:00.629033 329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:00.629936 320 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:00.640185 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32560
W20260812 20:00:00.641587 342 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:00.642540 342 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:00.651743 32713 master_runner.cc:429] Detected that this master 20d6a12c6f2b484991fd049c4273e503 is joining an existing cluster
I20260812 20:00:00.651850 32713 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:00:00.652329 32696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:00.653062 32713 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:00:00.653465 32713 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:00:00.653720 32713 tablet_replica.cc:333] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: stopping tablet replica
I20260812 20:00:00.656474 32713 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:36123
I20260812 20:00:00.722554 32456 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:36123: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20d6a12c6f2b484991fd049c4273e503"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 36123
}
attrs {
promote: true
}
}
dest_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc"
cas_config_opid_index: -1
I20260812 20:00:00.724428 32456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } }
I20260812 20:00:00.728117 32310 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:00:00.729220 32389 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:00:00.729386 32504 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:00:00.730527 358 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260812 20:00:00.731943 32440 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc -> Peer 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59:36123): Couldn't send request to peer 20d6a12c6f2b484991fd049c4273e503. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:00.736464 32504 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.744397 32310 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.750041 32389 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.761490 363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.762149 363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:00.762849 32504 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:36123
I20260812 20:00:00.764078 32713 master.cc:562] Master@127.25.14.59:36123 shutting down...
I20260812 20:00:00.766765 364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.768340 364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:00.769309 358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.769691 358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:00.775710 362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.776222 362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:00.775266 365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:00.776917 365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:00.784803 32713 master.cc:584] Master@127.25.14.59:36123 shutdown complete.
I20260812 20:00:00.784950 32713 master_runner.cc:306] Clearing existing system tablet
I20260812 20:00:00.785344 32713 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:00.785497 32713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:00.788990 32713 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:00:00.789273 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32560
I20260812 20:00:00.802139 368 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:00.804575 32713 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:00.805022 32713 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.000s sys 0.006s
I20260812 20:00:00.805188 32713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d6a12c6f2b484991fd049c4273e503"
format_stamp: "Formatted at 2026-08-12 20:00:00 on dist-test-slave-p3t5"
I20260812 20:00:00.810107 32713 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:00:00.816669 32713 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:00:00.816941 32713 log.cc:1199] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:00:00.817492 32713 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Deleting consensus metadata
I20260812 20:00:00.817803 32713 master_runner.cc:316] Copying system tablet from 127.25.14.60:37583
I20260812 20:00:00.828073 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:44125
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--webserver_port=34189
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123 with env {}
I20260812 20:00:00.849881 32713 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:37583
I20260812 20:00:00.858342 32479 tablet_copy_service.cc:140] P 6dadc22df3bb4263b425bf5d8ef0d1bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20d6a12c6f2b484991fd049c4273e503 ({username='slave'} at 127.0.0.1:38764)
I20260812 20:00:00.858705 32479 tablet_copy_service.cc:161] P 6dadc22df3bb4263b425bf5d8ef0d1bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20d6a12c6f2b484991fd049c4273e503 at {username='slave'} at 127.0.0.1:38764: session id = 20d6a12c6f2b484991fd049c4273e503-00000000000000000000000000000000
I20260812 20:00:00.863709 32479 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:00:00.866969 32713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:00:00.876730 32713 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: tablet copy: Starting download of 0 data blocks...
I20260812 20:00:00.877305 32713 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: tablet copy: Starting download of 1 WAL segments...
I20260812 20:00:00.881469 32713 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:00:00.886970 32479 tablet_copy_service.cc:342] P 6dadc22df3bb4263b425bf5d8ef0d1bc: Request end of tablet copy session 20d6a12c6f2b484991fd049c4273e503-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:38764
I20260812 20:00:00.887276 32479 tablet_copy_service.cc:434] P 6dadc22df3bb4263b425bf5d8ef0d1bc: ending tablet copy session 20d6a12c6f2b484991fd049c4273e503-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20d6a12c6f2b484991fd049c4273e503
I20260812 20:00:00.900307 32713 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:00.903374 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:00.903807 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:00.913128 383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:00.915755 32713 server_base.cc:1061] running on GCE node
I20260812 20:00:00.916253 32713 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:00.916395 32713 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:00.916467 32713 hybrid_clock.cc:648] HybridClock initialized: now 1786564800916442 us; error 0 us; skew 500 ppm
I20260812 20:00:00.916711 32713 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:00.918748 32713 webserver.cc:533] Webserver started at http://127.25.14.59:43137/ using document root <none> and password file <none>
I20260812 20:00:00.919144 32713 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:00.919250 32713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:00.922200 32713 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 20:00:00.924229 388 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:00.924739 32713 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:00:00.924890 32713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d6a12c6f2b484991fd049c4273e503"
format_stamp: "Formatted at 2026-08-12 20:00:00 on dist-test-slave-p3t5"
I20260812 20:00:00.925074 32713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:00.934969 32713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:00.935799 32713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:01.003302 32713 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:36123
I20260812 20:00:01.003329 445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:36123 every 8 connection(s)
I20260812 20:00:01.004534 32713 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:01.012092 446 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:01.017541 446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Bootstrap starting.
I20260812 20:00:01.033357 446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:01.033980 446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: Bootstrap complete.
I20260812 20:00:01.035146 446 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } }
I20260812 20:00:01.035444 446 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20d6a12c6f2b484991fd049c4273e503, State: Initialized, Role: LEARNER
I20260812 20:00:01.035804 446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } }
I20260812 20:00:01.036816 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:01.037201 446 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:01.037334 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:00:01.054090 464 catalog_manager.cc:1295] Loaded cluster ID: bb0dff71d65a41408b8ca70d7b1faa4d
I20260812 20:00:01.054240 464 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: loading cluster ID for follower catalog manager: success
I20260812 20:00:01.057093 464 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: acquiring CA information for follower catalog manager: success
I20260812 20:00:01.059401 464 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:01.295010 418 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:00:01.296067 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dadc22df3bb4263b425bf5d8ef0d1bc. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:01.296350 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:00:01.341883 374 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:01.342378 374 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:01.342511 374 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:01.352818 374 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:01.353019 374 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:01.364053 374 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:44125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34189
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:01.365955 374 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:01.367978 374 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:01.379737 472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:01.379719 471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:01.380787 474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:01.381175 374 server_base.cc:1061] running on GCE node
I20260812 20:00:01.381992 374 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:01.382179 374 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:01.382305 374 hybrid_clock.cc:648] HybridClock initialized: now 1786564801382277 us; error 0 us; skew 500 ppm
I20260812 20:00:01.382686 374 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:01.385241 374 webserver.cc:533] Webserver started at http://127.25.14.1:34189/ using document root <none> and password file <none>
I20260812 20:00:01.386220 374 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:01.386382 374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:01.392931 374 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 20:00:01.396522 480 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:01.398476 374 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:01.398696 374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 20:00:01.399526 374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:01.427217 374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:01.428272 374 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:01.428653 374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:01.429903 374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:01.431891 374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:01.431973 374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:01.432137 374 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:01.432206 374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:01.471299 374 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:44125
I20260812 20:00:01.471314 610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:44125 every 8 connection(s)
I20260812 20:00:01.473229 374 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:01.473601 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 374
I20260812 20:00:01.535254 611 heartbeater.cc:344] Connected to a master server at 127.25.14.59:36123
I20260812 20:00:01.535808 611 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:01.537321 611 heartbeater.cc:507] Master 127.25.14.59:36123 requested a full tablet report, sending...
I20260812 20:00:01.538553 615 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43631
I20260812 20:00:01.538717 615 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:01.539635 615 heartbeater.cc:507] Master 127.25.14.61:43631 requested a full tablet report, sending...
I20260812 20:00:01.540496 613 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37583
I20260812 20:00:01.540655 613 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:01.541245 613 heartbeater.cc:507] Master 127.25.14.60:37583 requested a full tablet report, sending...
I20260812 20:00:01.541831 614 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34197
I20260812 20:00:01.542014 614 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:01.542480 32379 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:01.542510 614 heartbeater.cc:507] Master 127.25.14.62:34197 requested a full tablet report, sending...
I20260812 20:00:01.542891 32456 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:01.544018 32456 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:39999
I20260812 20:00:01.544075 32298 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:01.544448 406 ts_manager.cc:194] Registered new tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:01.556830 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:01.570785 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:01.571012 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32265
I20260812 20:00:01.572301 32335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:01.690990 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32265
I20260812 20:00:01.721176 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34197
--webserver_interface=127.25.14.62
--webserver_port=44557
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123 with env {}
I20260812 20:00:01.750752 358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:01.751209 358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:01.752482 629 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc: attempting to promote NON_VOTER 20d6a12c6f2b484991fd049c4273e503 to VOTER
I20260812 20:00:01.753531 629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } }
I20260812 20:00:01.756520 418 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 1 LEARNER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:01.756552 32389 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Refusing update from remote peer 6dadc22df3bb4263b425bf5d8ef0d1bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260812 20:00:01.757125 32438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:34197: connect: Connection refused (error 111)
I20260812 20:00:01.757382 629 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:00:01.757783 630 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 20:00:01.767504 32438 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc -> Peer 5388a34d587143f9a6b1bdb1663f249f (127.25.14.62:34197): Couldn't send request to peer 5388a34d587143f9a6b1bdb1663f249f. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:34197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:01.768873 630 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.770033 418 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.773991 629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.774253 629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:01.774590 629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.774784 629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:01.774785 32389 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.775148 629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.775370 629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:01.775651 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.775919 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d6a12c6f2b484991fd049c4273e503 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:01.778869 639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:01.779214 639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4627e14c65429a851c75fdcb043ff0 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:00:02.154141 628 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:02.154644 628 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:02.154795 628 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:02.164269 628 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:02.164463 628 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:02.164582 628 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:02.164675 628 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:02.176157 628 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=44557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:02.178227 628 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:02.180310 628 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:02.190959 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:02.190747 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:02.191676 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:02.192690 628 server_base.cc:1061] running on GCE node
I20260812 20:00:02.193744 628 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:02.193938 628 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:02.194016 628 hybrid_clock.cc:648] HybridClock initialized: now 1786564802193996 us; error 0 us; skew 500 ppm
I20260812 20:00:02.194380 628 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:02.198071 628 webserver.cc:533] Webserver started at http://127.25.14.62:44557/ using document root <none> and password file <none>
I20260812 20:00:02.199311 628 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:02.199524 628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:02.206364 628 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:00:02.210641 654 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:02.212385 628 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:02.212641 628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5388a34d587143f9a6b1bdb1663f249f"
format_stamp: "Formatted at 2026-08-12 19:59:57 on dist-test-slave-p3t5"
I20260812 20:00:02.213536 628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:02.239562 628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:02.240664 628 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:02.241035 628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:02.264429 708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34197 every 8 connection(s)
I20260812 20:00:02.264395 628 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34197
I20260812 20:00:02.266778 628 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:00:02.275553 709 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:02.277276 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 628
I20260812 20:00:02.277665 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 374
I20260812 20:00:02.280704 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Bootstrap starting.
I20260812 20:00:02.288264 606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:02.307377 709 log.cc:826] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:02.321265 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:02.322057 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: Bootstrap complete.
I20260812 20:00:02.333169 709 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } }
I20260812 20:00:02.334283 709 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388a34d587143f9a6b1bdb1663f249f, State: Initialized, Role: FOLLOWER
I20260812 20:00:02.335333 709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } }
I20260812 20:00:02.336426 715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: true } } }
I20260812 20:00:02.336889 715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:02.337898 709 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:02.350193 727 catalog_manager.cc:1295] Loaded cluster ID: bb0dff71d65a41408b8ca70d7b1faa4d
I20260812 20:00:02.350342 727 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: loading cluster ID for follower catalog manager: success
I20260812 20:00:02.356298 727 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: acquiring CA information for follower catalog manager: success
I20260812 20:00:02.360414 727 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:02.421164 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 374
I20260812 20:00:02.448885 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:44125
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--webserver_port=34189
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123 with env {}
I20260812 20:00:02.761189 684 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d6a12c6f2b484991fd049c4273e503 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:02.769250 737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:02.769804 737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:02.770062 715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dadc22df3bb4263b425bf5d8ef0d1bc. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:02.770278 715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388a34d587143f9a6b1bdb1663f249f [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:00:02.886479 734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:02.886888 734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:02.887048 734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:02.896076 734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:02.896324 734 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:02.907292 734 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:44125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34189
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:02.909124 734 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:02.911319 734 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:02.922209 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:02.922674 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:02.923183 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:02.924770 734 server_base.cc:1061] running on GCE node
I20260812 20:00:02.925556 734 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:02.925845 734 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:02.925974 734 hybrid_clock.cc:648] HybridClock initialized: now 1786564802925932 us; error 0 us; skew 500 ppm
I20260812 20:00:02.926321 734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:02.928810 734 webserver.cc:533] Webserver started at http://127.25.14.1:34189/ using document root <none> and password file <none>
I20260812 20:00:02.929749 734 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:02.929862 734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:02.936231 734 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:02.939549 751 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:02.941213 734 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 20:00:02.941416 734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 20:00:02.942471 734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:02.975042 734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:02.976153 734 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:02.976485 734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:02.977589 734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:02.979514 734 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:02.979629 734 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:02.979792 734 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:02.979887 734 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:03.025368 734 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:44125
I20260812 20:00:03.025422 881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:44125 every 8 connection(s)
I20260812 20:00:03.027232 734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:03.028501 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 734
I20260812 20:00:03.070669 885 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34197
I20260812 20:00:03.071168 885 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:03.072315 885 heartbeater.cc:507] Master 127.25.14.62:34197 requested a full tablet report, sending...
I20260812 20:00:03.075320 672 ts_manager.cc:194] Registered new tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:03.090881 886 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43631
I20260812 20:00:03.091135 886 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:03.091813 886 heartbeater.cc:507] Master 127.25.14.61:43631 requested a full tablet report, sending...
I20260812 20:00:03.092919 882 heartbeater.cc:344] Connected to a master server at 127.25.14.59:36123
I20260812 20:00:03.093063 883 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37583
I20260812 20:00:03.093237 883 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:03.093847 883 heartbeater.cc:507] Master 127.25.14.60:37583 requested a full tablet report, sending...
I20260812 20:00:03.093088 882 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:03.094152 32379 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:03.094712 882 heartbeater.cc:507] Master 127.25.14.59:36123 requested a full tablet report, sending...
I20260812 20:00:03.095484 32456 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:03.096190 406 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:03.096601 32456 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:56813
I20260812 20:00:03.097226 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 734
I20260812 20:00:03.110407 877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:03.203094 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 734
I20260812 20:00:03.230113 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:44125
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--webserver_port=34189
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--fault_before_start=1.0 with env {}
I20260812 20:00:03.289713 638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5388a34d587143f9a6b1bdb1663f249f" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34197 } } peers { permanent_uuid: "3f4627e14c65429a851c75fdcb043ff0" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 43631 } } peers { permanent_uuid: "6dadc22df3bb4263b425bf5d8ef0d1bc" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37583 } } peers { permanent_uuid: "20d6a12c6f2b484991fd049c4273e503" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 36123 } attrs { promote: false } } }
I20260812 20:00:03.290141 638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc [sys.catalog]: This master's current role is: LEADER
W20260812 20:00:03.633347 901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:03.633836 901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:03.633991 901 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:00:03.634088 901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:03.643887 901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:03.644230 901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:03.655265 901 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:44125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34189
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:03.657147 901 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:03.659191 901 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:03.671386 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:03.671630 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:03.671380 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:03.672394 901 server_base.cc:1061] running on GCE node
I20260812 20:00:03.673084 901 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:03.673270 901 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:03.673413 901 hybrid_clock.cc:648] HybridClock initialized: now 1786564803673345 us; error 0 us; skew 500 ppm
I20260812 20:00:03.673831 901 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:03.676275 901 webserver.cc:533] Webserver started at http://127.25.14.1:34189/ using document root <none> and password file <none>
I20260812 20:00:03.677137 901 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:03.677291 901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:03.683892 901 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:03.687549 919 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:03.689479 901 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:03.689801 901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 20:00:03.690869 901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:03.708272 901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:03.709271 901 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:03.709579 901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:03.710839 901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:03.713047 901 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:03.713145 901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:03.713301 901 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:03.713390 901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:00:03.713872 901 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:00:03.713963 901 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:00:03.714051 901 fault_injection.cc:59] injecting fault for kudu (pid 901): FLAGS_fault_before_start (process will exit)
I20260812 20:00:03.743455 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:44125
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583
--webserver_port=34189
--webserver_interface=127.25.14.1
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123 with env {}
W20260812 20:00:04.150010 928 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:04.150447 928 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:04.150602 928 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:04.159526 928 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:04.159796 928 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:04.170723 928 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:44125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=34189
--tserver_master_addrs=127.25.14.62:34197,127.25.14.61:43631,127.25.14.60:37583,127.25.14.59:36123
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:04.172667 928 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:04.175058 928 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:04.186671 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:04.186671 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:04.186592 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:04.187716 928 server_base.cc:1061] running on GCE node
I20260812 20:00:04.188445 928 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:00:04.188652 928 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:00:04.188781 928 hybrid_clock.cc:648] HybridClock initialized: now 1786564804188756 us; error 0 us; skew 500 ppm
I20260812 20:00:04.189150 928 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:04.191838 928 webserver.cc:533] Webserver started at http://127.25.14.1:34189/ using document root <none> and password file <none>
I20260812 20:00:04.192773 928 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:04.192888 928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:04.199863 928 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 20:00:04.203856 947 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:04.205869 928 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:04.206153 928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "8096f2c3cfed4fd4b4b7570cf41094ed"
format_stamp: "Formatted at 2026-08-12 19:59:59 on dist-test-slave-p3t5"
I20260812 20:00:04.207023 928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:04.236914 928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:04.237985 928 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:04.238422 928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:04.239696 928 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:04.241854 928 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:04.241937 928 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:04.242136 928 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:04.242192 928 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:04.281301 928 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:44125
I20260812 20:00:04.281340 1077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:44125 every 8 connection(s)
I20260812 20:00:04.283195 928 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:04.286314 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 928
I20260812 20:00:04.286671 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 928
I20260812 20:00:04.313983 1078 heartbeater.cc:344] Connected to a master server at 127.25.14.59:36123
I20260812 20:00:04.314466 1078 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:04.314781 1081 heartbeater.cc:344] Connected to a master server at 127.25.14.61:43631
I20260812 20:00:04.314931 1081 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:04.316282 1081 heartbeater.cc:507] Master 127.25.14.61:43631 requested a full tablet report, sending...
I20260812 20:00:04.317451 1072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:04.413151 1078 heartbeater.cc:507] Master 127.25.14.59:36123 requested a full tablet report, sending...
I20260812 20:00:04.413270 32379 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:04.413796 1080 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34197
I20260812 20:00:04.413987 1080 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:04.414595 1080 heartbeater.cc:507] Master 127.25.14.62:34197 requested a full tablet report, sending...
I20260812 20:00:04.415431 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 928
I20260812 20:00:04.416273 406 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
I20260812 20:00:04.419849 672 ts_manager.cc:194] Re-registered known tserver with Master: 8096f2c3cfed4fd4b4b7570cf41094ed (127.25.14.1:44125)
W20260812 20:00:04.441887 392 connection.cc:573] server connection from 127.25.14.1:36781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:04.442049 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 628
W20260812 20:00:04.442058 656 connection.cc:573] server connection from 127.25.14.1:38179 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:04.449090 704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:04.557737 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 628
I20260812 20:00:04.590346 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32345
I20260812 20:00:04.591637 32411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:04.703317 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32345
W20260812 20:00:04.735608 32439 connection.cc:573] client connection to 127.25.14.61:43631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 20:00:04.735877 32438 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc -> Peer 5388a34d587143f9a6b1bdb1663f249f (127.25.14.62:34197): Couldn't send request to peer 5388a34d587143f9a6b1bdb1663f249f. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:34197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:04.735914 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32420
W20260812 20:00:04.736927 32439 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6dadc22df3bb4263b425bf5d8ef0d1bc -> Peer 3f4627e14c65429a851c75fdcb043ff0 (127.25.14.61:43631): Couldn't send request to peer 3f4627e14c65429a851c75fdcb043ff0. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.61:43631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:04.737243 32491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:04.854686 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32420
W20260812 20:00:04.883746 392 connection.cc:573] server connection from 127.0.0.1:46292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:04.884202 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 32713
I20260812 20:00:04.885596 440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:04.999249 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 32713
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/3 (24855 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/4
2026-08-12T20:00:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:05Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 20:00:13: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 20:00:22.308383 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41971
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235 with env {}
W20260812 20:00:22.697074 1425 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:22.697436 1425 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:22.697506 1425 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:22.706283 1425 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:22.706398 1425 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:22.706456 1425 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:22.706504 1425 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:22.716846 1425 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:22.718658 1425 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:22.720644 1425 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:22.730041 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:22.730039 1433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:22.730096 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:22.731077 1425 server_base.cc:1061] running on GCE node
I20260812 20:00:22.731870 1425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:22.733366 1425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:22.734578 1425 hybrid_clock.cc:648] HybridClock initialized: now 1786564822734511 us; error 56 us; skew 500 ppm
I20260812 20:00:22.734985 1425 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:22.737435 1425 webserver.cc:533] Webserver started at http://127.25.14.62:44133/ using document root <none> and password file <none>
I20260812 20:00:22.738401 1425 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:22.738530 1425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:22.738951 1425 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:22.741593 1425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "68828397ae4e4c9386f67192df91d950"
format_stamp: "Formatted at 2026-08-12 20:00:22 on dist-test-slave-p3t5"
I20260812 20:00:22.742573 1425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "68828397ae4e4c9386f67192df91d950"
format_stamp: "Formatted at 2026-08-12 20:00:22 on dist-test-slave-p3t5"
I20260812 20:00:22.748608 1425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260812 20:00:22.752588 1440 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:22.754207 1425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:00:22.754441 1425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "68828397ae4e4c9386f67192df91d950"
format_stamp: "Formatted at 2026-08-12 20:00:22 on dist-test-slave-p3t5"
I20260812 20:00:22.754698 1425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:22.774871 1425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:22.775928 1425 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:22.776273 1425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:22.799573 1425 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41971
I20260812 20:00:22.799600 1499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41971 every 8 connection(s)
I20260812 20:00:22.801451 1425 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:00:22.806492 1500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:22.809713 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1425
I20260812 20:00:22.809949 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:00:22.810997 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:37727
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235 with env {}
I20260812 20:00:22.813050 1500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:22.837126 1500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:22.838482 1441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
W20260812 20:00:22.840464 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37727: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
I20260812 20:00:22.889938 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } attempt: 1
W20260812 20:00:22.892464 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37727: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
I20260812 20:00:22.960800 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } attempt: 2
W20260812 20:00:22.963021 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37727: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
I20260812 20:00:23.054361 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } attempt: 3
W20260812 20:00:23.056775 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37727: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
I20260812 20:00:23.200116 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } attempt: 4
W20260812 20:00:23.201860 1504 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:23.202361 1504 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:23.202523 1504 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:23.202664 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37727: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37727: connect: Connection refused (error 111)
W20260812 20:00:23.211647 1504 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:23.211771 1504 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:23.211872 1504 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:23.212001 1504 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:23.223155 1504 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:23.225081 1504 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:23.227200 1504 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:23.236987 1511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:23.237270 1512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:23.237459 1514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:23.238873 1504 server_base.cc:1061] running on GCE node
I20260812 20:00:23.239773 1504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:23.241374 1504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:23.242688 1504 hybrid_clock.cc:648] HybridClock initialized: now 1786564823242600 us; error 102 us; skew 500 ppm
I20260812 20:00:23.243129 1504 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:23.245697 1504 webserver.cc:533] Webserver started at http://127.25.14.61:35651/ using document root <none> and password file <none>
I20260812 20:00:23.246562 1504 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:23.246722 1504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:23.247115 1504 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:23.251770 1504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "8d37053881924479a72dabeeb6eb7f8a"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.252767 1504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "8d37053881924479a72dabeeb6eb7f8a"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.259102 1504 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 20:00:23.263275 1521 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:23.265033 1504 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:23.265215 1504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "8d37053881924479a72dabeeb6eb7f8a"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.265419 1504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:23.286372 1504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:23.287390 1504 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:23.287665 1504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:23.311012 1504 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:37727
I20260812 20:00:23.311033 1583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:37727 every 8 connection(s)
I20260812 20:00:23.312937 1504 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:00:23.315410 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1504
I20260812 20:00:23.315618 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:00:23.316700 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40235
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235 with env {}
I20260812 20:00:23.318307 1584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:23.324765 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:23.341604 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:23.353345 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:23.354908 1522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
W20260812 20:00:23.356889 1584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.406297 1584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 1
W20260812 20:00:23.408739 1584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.477012 1584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 2
W20260812 20:00:23.479400 1584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.502058 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } attempt: 5
I20260812 20:00:23.510059 1500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:23.512187 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.563549 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 1
W20260812 20:00:23.565814 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.570685 1584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 3
W20260812 20:00:23.572953 1584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.634088 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 2
W20260812 20:00:23.636624 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.716238 1584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 4
W20260812 20:00:23.719045 1584 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
W20260812 20:00:23.727118 1587 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:23.727532 1587 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:23.727679 1587 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:23.736593 1587 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:23.736701 1587 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:23.736760 1587 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:23.736802 1587 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:23.742971 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 3
W20260812 20:00:23.745170 1500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:40235: Network error: Client connection negotiation failed: client connection to 127.25.14.60:40235: connect: Connection refused (error 111)
I20260812 20:00:23.748399 1587 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:23.750270 1587 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:23.752683 1587 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:23.761529 1595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:23.761601 1596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:23.762673 1600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:23.763271 1587 server_base.cc:1061] running on GCE node
I20260812 20:00:23.764195 1587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:23.765911 1587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:23.767174 1587 hybrid_clock.cc:648] HybridClock initialized: now 1786564823767097 us; error 62 us; skew 500 ppm
I20260812 20:00:23.767524 1587 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:23.769834 1587 webserver.cc:533] Webserver started at http://127.25.14.60:37977/ using document root <none> and password file <none>
I20260812 20:00:23.770624 1587 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:23.770722 1587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:23.771068 1587 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:23.773836 1587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "176f25e4cfdd41c08e9a3a8afc8f03da"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.774729 1587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "176f25e4cfdd41c08e9a3a8afc8f03da"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.780632 1587 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 20:00:23.784699 1607 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:23.786300 1587 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:00:23.786520 1587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "176f25e4cfdd41c08e9a3a8afc8f03da"
format_stamp: "Formatted at 2026-08-12 20:00:23 on dist-test-slave-p3t5"
I20260812 20:00:23.786777 1587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:23.804620 1587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:23.805721 1587 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:23.806051 1587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:23.828537 1587 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:40235
I20260812 20:00:23.828558 1665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:40235 every 8 connection(s)
I20260812 20:00:23.830488 1587 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:00:23.834613 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1587
I20260812 20:00:23.834862 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:00:23.835881 1666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:23.844532 1666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:23.859311 1666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:23.866164 1666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:23.877651 1666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Bootstrap starting.
I20260812 20:00:23.881054 1666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:23.882344 1666 log.cc:826] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:23.885259 1666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: No bootstrap required, opened a new log
I20260812 20:00:23.890959 1666 raft_consensus.cc:359] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:23.891309 1666 raft_consensus.cc:385] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:23.891443 1666 raft_consensus.cc:740] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176f25e4cfdd41c08e9a3a8afc8f03da, State: Initialized, Role: FOLLOWER
I20260812 20:00:23.892150 1666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:23.893265 1673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:23.893680 1673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:23.894742 1666 sys_catalog.cc:565] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:00:23.904652 1685 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:23.904821 1685 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:23.902509 1587 master_runner.cc:187] Error getting master registration for 127.25.14.62:41971: OK, instance_id { permanent_uuid: "68828397ae4e4c9386f67192df91d950" instance_seqno: 1786564822793230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:00:23.908824 1587 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:00:23.922418 1500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 4
I20260812 20:00:23.933378 1500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Bootstrap starting.
I20260812 20:00:23.936961 1500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:23.938277 1500 log.cc:826] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:23.942498 1500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: No bootstrap required, opened a new log
I20260812 20:00:23.947973 1500 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:23.948241 1500 raft_consensus.cc:385] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:23.948431 1500 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68828397ae4e4c9386f67192df91d950, State: Initialized, Role: FOLLOWER
I20260812 20:00:23.949148 1500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:23.950307 1688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:23.950631 1688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:23.951570 1500 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:00:23.965397 1700 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:23.965692 1700 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:23.963771 1425 master_runner.cc:187] Error getting master registration for 127.25.14.61:37727: OK, instance_id { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" instance_seqno: 1786564823304805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:00:23.970847 1425 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:00:24.018357 1584 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } attempt: 5
W20260812 20:00:24.029882 1449 tablet.cc:2406] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:00:24.032989 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: Bootstrap starting.
I20260812 20:00:24.037669 1584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:24.039206 1584 log.cc:826] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:24.043524 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: No bootstrap required, opened a new log
I20260812 20:00:24.052973 1584 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:24.053385 1584 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:24.053530 1584 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d37053881924479a72dabeeb6eb7f8a, State: Initialized, Role: FOLLOWER
I20260812 20:00:24.054476 1584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
W20260812 20:00:24.058558 1614 tablet.cc:2406] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:00:24.058434 1703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.058801 1703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:24.061324 1584 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:24.088490 1504 master_runner.cc:264] No leader master found from master 8d37053881924479a72dabeeb6eb7f8a
I20260812 20:00:24.088665 1504 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 20:00:24.092217 1718 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:24.092567 1718 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:24.254875 1673 raft_consensus.cc:493] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:00:24.255230 1673 raft_consensus.cc:515] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:24.257493 1673 leader_election.cc:290] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68828397ae4e4c9386f67192df91d950 (127.25.14.62:41971), 8d37053881924479a72dabeeb6eb7f8a (127.25.14.61:37727)
I20260812 20:00:24.258544 1473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68828397ae4e4c9386f67192df91d950" is_pre_election: true
I20260812 20:00:24.258802 1555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d37053881924479a72dabeeb6eb7f8a" is_pre_election: true
I20260812 20:00:24.259148 1473 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176f25e4cfdd41c08e9a3a8afc8f03da in term 0.
I20260812 20:00:24.259433 1555 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176f25e4cfdd41c08e9a3a8afc8f03da in term 0.
I20260812 20:00:24.259922 1612 leader_election.cc:304] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176f25e4cfdd41c08e9a3a8afc8f03da, 68828397ae4e4c9386f67192df91d950; no voters:
I20260812 20:00:24.261394 1673 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:00:24.261713 1673 raft_consensus.cc:493] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:00:24.261948 1673 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:24.266055 1673 raft_consensus.cc:515] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
I20260812 20:00:24.267864 1473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68828397ae4e4c9386f67192df91d950"
I20260812 20:00:24.268177 1473 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:24.272858 1473 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176f25e4cfdd41c08e9a3a8afc8f03da in term 1.
I20260812 20:00:24.273792 1612 leader_election.cc:304] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176f25e4cfdd41c08e9a3a8afc8f03da, 68828397ae4e4c9386f67192df91d950; no voters:
I20260812 20:00:24.279192 1555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d37053881924479a72dabeeb6eb7f8a"
I20260812 20:00:24.279548 1555 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:24.283943 1555 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176f25e4cfdd41c08e9a3a8afc8f03da in term 1.
I20260812 20:00:24.285235 1673 leader_election.cc:290] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [CANDIDATE]: Term 1 election: Requested vote from peers 68828397ae4e4c9386f67192df91d950 (127.25.14.62:41971), 8d37053881924479a72dabeeb6eb7f8a (127.25.14.61:37727)
I20260812 20:00:24.285873 1673 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:00:24.286475 1673 raft_consensus.cc:697] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 1 LEADER]: Becoming Leader. State: Replica: 176f25e4cfdd41c08e9a3a8afc8f03da, State: Running, Role: LEADER
I20260812 20:00:24.287542 1673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } }
W20260812 20:00:24.290956 1530 tablet.cc:2406] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:00:24.315156 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176f25e4cfdd41c08e9a3a8afc8f03da. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.315599 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:24.326656 1722 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:00:24.329843 1722 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:00:24.347383 1555 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:00:24.348913 1720 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:00:24.357929 1473 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:00:24.359309 1720 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:00:24.396967 1688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176f25e4cfdd41c08e9a3a8afc8f03da. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.397305 1688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:24.400626 1723 mvcc.cc:204] Tried to move back new op lower bound from 7317769520512557056 to 7317769520303747072. Current Snapshot: MvccSnapshot[applied={T|T < 7317769520512557056}]
I20260812 20:00:24.406080 1673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.406359 1673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:24.408708 1722 catalog_manager.cc:1383] Generated new cluster ID: 28a701bb84404b98b31eed303b643527
I20260812 20:00:24.408825 1722 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:00:24.414253 1703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176f25e4cfdd41c08e9a3a8afc8f03da. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.414529 1703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:24.417647 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.418637 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:24.419952 1688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.420193 1688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:24.432703 1725 mvcc.cc:204] Tried to move back new op lower bound from 7317769520512557056 to 7317769520303747072. Current Snapshot: MvccSnapshot[applied={T|T < 7317769520512557056 or (T in {7317769520512557056})}]
I20260812 20:00:24.434197 1703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } }
I20260812 20:00:24.434448 1703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:24.437491 1724 mvcc.cc:204] Tried to move back new op lower bound from 7317769520512557056 to 7317769520303747072. Current Snapshot: MvccSnapshot[applied={T|T < 7317769520512557056}]
I20260812 20:00:24.460662 1722 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:00:24.462919 1722 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:00:24.488515 1722 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Generated new TSK 0
I20260812 20:00:24.489898 1722 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 20:00:24.518115 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 20:00:24.950384 1734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:24.950773 1734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:24.950953 1734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:24.961208 1734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:24.961444 1734 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:24.968125 1700 catalog_manager.cc:1295] Loaded cluster ID: 28a701bb84404b98b31eed303b643527
I20260812 20:00:24.968286 1700 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: loading cluster ID for follower catalog manager: success
I20260812 20:00:24.971753 1700 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: acquiring CA information for follower catalog manager: success
I20260812 20:00:24.973408 1734 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:24.973966 1700 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:24.975219 1734 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:24.977226 1734 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:24.988675 1743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:24.988692 1747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:24.988649 1745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:24.989568 1734 server_base.cc:1061] running on GCE node
I20260812 20:00:24.990217 1734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:24.991235 1734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:24.992483 1734 hybrid_clock.cc:648] HybridClock initialized: now 1786564824992439 us; error 38 us; skew 500 ppm
I20260812 20:00:24.992867 1734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:24.995268 1734 webserver.cc:533] Webserver started at http://127.25.14.1:35999/ using document root <none> and password file <none>
I20260812 20:00:24.996172 1734 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:24.996297 1734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:24.996905 1734 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:24.999712 1734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:25.000592 1734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:25.006464 1734 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 20:00:25.010654 1754 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.012176 1734 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260812 20:00:25.012382 1734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:25.012652 1734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:25.044036 1734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:25.045076 1734 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:25.045408 1734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:25.046555 1734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:25.048784 1734 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:25.048894 1734 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.049089 1734 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:25.049211 1734 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.093149 1734 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40889
I20260812 20:00:25.093276 1878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40889 every 8 connection(s)
I20260812 20:00:25.095033 1734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:25.095103 1718 catalog_manager.cc:1295] Loaded cluster ID: 28a701bb84404b98b31eed303b643527
I20260812 20:00:25.095211 1718 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: loading cluster ID for follower catalog manager: success
I20260812 20:00:25.099552 1718 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: acquiring CA information for follower catalog manager: success
I20260812 20:00:25.105671 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1734
I20260812 20:00:25.105938 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 20:00:25.107937 1718 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:25.119423 1880 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41971
I20260812 20:00:25.119952 1880 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:25.121762 1880 heartbeater.cc:507] Master 127.25.14.62:41971 requested a full tablet report, sending...
I20260812 20:00:25.122238 1879 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40235
I20260812 20:00:25.122402 1879 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:25.123091 1879 heartbeater.cc:507] Master 127.25.14.60:40235 requested a full tablet report, sending...
I20260812 20:00:25.124843 1881 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37727
I20260812 20:00:25.125005 1881 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:25.125370 1463 ts_manager.cc:194] Registered new tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:25.125422 1881 heartbeater.cc:507] Master 127.25.14.61:37727 requested a full tablet report, sending...
I20260812 20:00:25.127343 1628 ts_manager.cc:194] Registered new tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:25.127519 1542 ts_manager.cc:194] Registered new tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:25.129257 1628 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:58409
I20260812 20:00:25.137493 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:25.140108 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:40437
--webserver_interface=127.25.14.59
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437 with env {}
W20260812 20:00:25.545779 1890 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:25.546286 1890 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:25.546452 1890 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:25.558926 1890 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:25.559082 1890 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:25.559243 1890 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:25.559329 1890 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:25.575184 1890 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:25.577786 1890 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:25.580883 1890 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:25.593327 1896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:25.593861 1897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:25.597514 1890 server_base.cc:1061] running on GCE node
W20260812 20:00:25.598341 1899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:25.599685 1890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:25.601874 1890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:25.603250 1890 hybrid_clock.cc:648] HybridClock initialized: now 1786564825603165 us; error 60 us; skew 500 ppm
I20260812 20:00:25.603739 1890 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:25.606843 1890 webserver.cc:533] Webserver started at http://127.25.14.59:46845/ using document root <none> and password file <none>
I20260812 20:00:25.608112 1890 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:25.608280 1890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:25.608810 1890 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:25.612818 1890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
format_stamp: "Formatted at 2026-08-12 20:00:25 on dist-test-slave-p3t5"
I20260812 20:00:25.614255 1890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
format_stamp: "Formatted at 2026-08-12 20:00:25 on dist-test-slave-p3t5"
I20260812 20:00:25.623413 1890 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260812 20:00:25.629433 1905 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.631829 1890 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 20:00:25.632087 1890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
format_stamp: "Formatted at 2026-08-12 20:00:25 on dist-test-slave-p3t5"
I20260812 20:00:25.632398 1890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:25.656430 1890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:25.657531 1890 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:25.657878 1890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:25.681248 1890 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:40437
I20260812 20:00:25.681228 1966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:40437 every 8 connection(s)
I20260812 20:00:25.683122 1890 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:25.685215 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 1890
I20260812 20:00:25.689981 1967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:25.698343 1967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:25.709189 1967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:25.716634 1967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:25.724114 1967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:25.735340 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Bootstrap starting.
I20260812 20:00:25.739081 1967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:25.740487 1967 log.cc:826] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:25.743917 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: No bootstrap required, opened a new log
I20260812 20:00:25.750196 1967 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } }
I20260812 20:00:25.750658 1967 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:25.750821 1967 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee06fae2293f47a89a3a47e7f2f1e650, State: Initialized, Role: FOLLOWER
I20260812 20:00:25.751652 1967 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } }
I20260812 20:00:25.752698 1978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } } }
I20260812 20:00:25.753288 1978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:25.754340 1967 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:00:25.766461 1992 catalog_manager.cc:1594] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:25.766646 1992 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:25.776225 1890 master_runner.cc:429] Detected that this master ee06fae2293f47a89a3a47e7f2f1e650 is joining an existing cluster
I20260812 20:00:25.776401 1890 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:00:25.777839 1890 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:00:25.778258 1890 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:00:25.778523 1890 tablet_replica.cc:333] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: stopping tablet replica
I20260812 20:00:25.780325 1890 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:40437
I20260812 20:00:25.835253 1627 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:40437: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 40437
}
attrs {
promote: true
}
}
dest_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da"
cas_config_opid_index: -1
I20260812 20:00:25.837319 1627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } }
I20260812 20:00:25.841295 1472 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:00:25.842195 1610 consensus_peers.cc:597] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da -> Peer ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59:40437): Couldn't send request to peer ee06fae2293f47a89a3a47e7f2f1e650. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:25.842509 1552 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:00:25.842437 2005 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:00:25.843248 1731 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:00:25.846596 2005 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.847483 1472 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.848528 1552 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.851719 2005 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:40437
I20260812 20:00:25.852916 1890 master.cc:562] Master@127.25.14.59:40437 shutting down...
I20260812 20:00:25.854409 2010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.854755 2010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:25.862862 1731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.863201 1731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:25.862556 2011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.863786 2011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:25.864603 2005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.864562 1731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:25.864818 2005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:25.864928 1731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
W20260812 20:00:25.900040 1890 connection.cc:466] server connection from 127.0.0.1:47048 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:47048 (request call id 58) could send its response
I20260812 20:00:25.902300 1890 master.cc:584] Master@127.25.14.59:40437 shutdown complete.
I20260812 20:00:25.902403 1890 master_runner.cc:306] Clearing existing system tablet
I20260812 20:00:25.902908 1890 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:25.903018 1890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:25.906196 1890 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260812 20:00:25.908075 2015 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.908514 1890 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:25.908841 1890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:00:25.908933 1890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
format_stamp: "Formatted at 2026-08-12 20:00:25 on dist-test-slave-p3t5"
I20260812 20:00:25.912619 1890 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:00:25.918929 1890 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:00:25.919160 1890 log.cc:1199] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:00:25.919660 1890 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Deleting consensus metadata
I20260812 20:00:25.919948 1890 master_runner.cc:316] Copying system tablet from 127.25.14.60:40235
I20260812 20:00:25.923552 1890 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:40235
I20260812 20:00:25.931679 1648 tablet_copy_service.cc:140] P 176f25e4cfdd41c08e9a3a8afc8f03da: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee06fae2293f47a89a3a47e7f2f1e650 ({username='slave'} at 127.0.0.1:57562)
I20260812 20:00:25.932003 1648 tablet_copy_service.cc:161] P 176f25e4cfdd41c08e9a3a8afc8f03da: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee06fae2293f47a89a3a47e7f2f1e650 at {username='slave'} at 127.0.0.1:57562: session id = ee06fae2293f47a89a3a47e7f2f1e650-00000000000000000000000000000000
I20260812 20:00:25.935247 1648 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:00:25.938184 1890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:00:25.943598 1890 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: tablet copy: Starting download of 0 data blocks...
I20260812 20:00:25.943976 1890 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: tablet copy: Starting download of 1 WAL segments...
I20260812 20:00:25.947366 1890 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:00:25.951331 1648 tablet_copy_service.cc:342] P 176f25e4cfdd41c08e9a3a8afc8f03da: Request end of tablet copy session ee06fae2293f47a89a3a47e7f2f1e650-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:57562
I20260812 20:00:25.951646 1648 tablet_copy_service.cc:434] P 176f25e4cfdd41c08e9a3a8afc8f03da: ending tablet copy session ee06fae2293f47a89a3a47e7f2f1e650-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee06fae2293f47a89a3a47e7f2f1e650
W20260812 20:00:25.959018 1890 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:00:25.963292 1890 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:25.966100 2024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:25.967859 2026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:25.968168 2028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:25.971017 1890 server_base.cc:1061] running on GCE node
I20260812 20:00:25.971393 1890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:25.972179 1890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:25.973393 1890 hybrid_clock.cc:648] HybridClock initialized: now 1786564825973329 us; error 56 us; skew 500 ppm
I20260812 20:00:25.973726 1890 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:25.975390 1890 webserver.cc:533] Webserver started at http://127.25.14.59:34735/ using document root <none> and password file <none>
I20260812 20:00:25.975740 1890 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:25.975859 1890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:25.978359 1890 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 20:00:25.980304 2034 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:25.980827 1890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 20:00:25.980975 1890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "ee06fae2293f47a89a3a47e7f2f1e650"
format_stamp: "Formatted at 2026-08-12 20:00:25 on dist-test-slave-p3t5"
I20260812 20:00:25.981150 1890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:25.991875 1890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:25.992709 1890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:26.013532 1890 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:40437
I20260812 20:00:26.013572 2091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:40437 every 8 connection(s)
I20260812 20:00:26.014626 1890 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:26.018615 2092 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:26.021057 2092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Bootstrap starting.
I20260812 20:00:26.036589 2092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:26.037187 2092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: Bootstrap complete.
I20260812 20:00:26.038424 2092 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } }
I20260812 20:00:26.038735 2092 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee06fae2293f47a89a3a47e7f2f1e650, State: Initialized, Role: LEARNER
I20260812 20:00:26.039018 2092 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } }
I20260812 20:00:26.039985 2095 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:26.040319 2095 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:00:26.040409 2092 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:26.049535 2108 catalog_manager.cc:1295] Loaded cluster ID: 28a701bb84404b98b31eed303b643527
I20260812 20:00:26.049721 2108 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: loading cluster ID for follower catalog manager: success
I20260812 20:00:26.052762 2108 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: acquiring CA information for follower catalog manager: success
I20260812 20:00:26.056150 2108 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:26.062704 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1734
I20260812 20:00:26.072716 1874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:26.174350 1879 heartbeater.cc:499] Master 127.25.14.60:40235 was elected leader, sending a full tablet report...
I20260812 20:00:26.175591 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1734
W20260812 20:00:26.202633 1443 connection.cc:573] server connection from 127.25.14.1:45617 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:26.203922 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40889
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--webserver_port=35999
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437 with env {}
I20260812 20:00:26.279006 2063 raft_consensus.cc:1217] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:00:26.280107 2095 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176f25e4cfdd41c08e9a3a8afc8f03da. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:26.280418 2095 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:00:26.612365 2112 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:26.612735 2112 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:26.612928 2112 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:26.621824 2112 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:26.622052 2112 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:26.632608 2112 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35999
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:26.634521 2112 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:26.636576 2112 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:26.648516 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:26.648586 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:26.648556 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:26.650074 2112 server_base.cc:1061] running on GCE node
I20260812 20:00:26.650784 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:26.651895 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:26.653165 2112 hybrid_clock.cc:648] HybridClock initialized: now 1786564826653090 us; error 71 us; skew 500 ppm
I20260812 20:00:26.653540 2112 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:26.656204 2112 webserver.cc:533] Webserver started at http://127.25.14.1:35999/ using document root <none> and password file <none>
I20260812 20:00:26.657078 2112 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:26.657248 2112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:26.663801 2112 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:26.667155 2127 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:26.668722 2112 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:00:26.668937 2112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:26.669870 2112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:26.686264 2112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:26.687325 2112 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:26.687680 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:26.688772 2112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:26.690774 2112 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:26.690893 2112 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:26.691064 2112 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:26.691141 2112 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:26.729193 2112 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40889
I20260812 20:00:26.729233 2259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40889 every 8 connection(s)
I20260812 20:00:26.730940 2112 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:26.737917 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2112
I20260812 20:00:26.771595 2262 heartbeater.cc:344] Connected to a master server at 127.25.14.59:40437
I20260812 20:00:26.771672 2261 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41971
I20260812 20:00:26.771785 2260 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40235
I20260812 20:00:26.772121 2261 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:26.773449 2261 heartbeater.cc:507] Master 127.25.14.62:41971 requested a full tablet report, sending...
I20260812 20:00:26.772115 2262 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:26.772120 2260 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:26.775117 2260 heartbeater.cc:507] Master 127.25.14.60:40235 requested a full tablet report, sending...
I20260812 20:00:26.775136 2262 heartbeater.cc:507] Master 127.25.14.59:40437 requested a full tablet report, sending...
I20260812 20:00:26.775911 1463 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:26.776616 1627 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:26.777138 2053 ts_manager.cc:194] Registered new tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:26.777850 1627 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:45803
I20260812 20:00:26.781162 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:26.782224 2263 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37727
I20260812 20:00:26.782450 2263 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:26.782804 2263 heartbeater.cc:507] Master 127.25.14.61:37727 requested a full tablet report, sending...
I20260812 20:00:26.784427 1542 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:26.790751 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:26.790948 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1425
I20260812 20:00:26.792021 1495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:26.835163 1731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:26.835491 1731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:26.848044 2276 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da: attempting to promote NON_VOTER ee06fae2293f47a89a3a47e7f2f1e650 to VOTER
I20260812 20:00:26.849224 2276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } }
I20260812 20:00:26.852015 2063 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 1 LEARNER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:26.852363 1552 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:26.852893 1731 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:00:26.853245 2276 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:00:26.857713 1731 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.858871 2063 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.862128 2276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.862432 2276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:26.862504 2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.862779 2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:26.863251 2278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.863626 2278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:26.863360 1552 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.865407 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.866127 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06fae2293f47a89a3a47e7f2f1e650 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:26.867762 2281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:26.868083 2281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d37053881924479a72dabeeb6eb7f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:26.918083 1472 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Refusing update from remote peer 176f25e4cfdd41c08e9a3a8afc8f03da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:26.918794 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1425
I20260812 20:00:26.919075 2276 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [LEADER]: Connected to new peer: Peer: permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20260812 20:00:26.949195 1612 connection.cc:573] client connection to 127.25.14.62:41971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 20:00:26.949548 1612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:26.950119 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41971
--webserver_interface=127.25.14.62
--webserver_port=44133
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437 with env {}
W20260812 20:00:26.952154 1612 consensus_peers.cc:597] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da -> Peer 68828397ae4e4c9386f67192df91d950 (127.25.14.62:41971): Couldn't send request to peer 68828397ae4e4c9386f67192df91d950. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:00:27.349042 2286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:27.349442 2286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:27.349529 2286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:27.358939 2286 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:27.359050 2286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:27.359108 2286 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:27.359158 2286 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:27.370816 2286 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=44133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:27.372815 2286 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:27.374908 2286 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:27.385007 2294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:27.385035 2295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:27.385382 2297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:27.386480 2286 server_base.cc:1061] running on GCE node
I20260812 20:00:27.387534 2286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:27.389333 2286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:27.390564 2286 hybrid_clock.cc:648] HybridClock initialized: now 1786564827390466 us; error 89 us; skew 500 ppm
I20260812 20:00:27.390990 2286 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:27.394095 2286 webserver.cc:533] Webserver started at http://127.25.14.62:44133/ using document root <none> and password file <none>
I20260812 20:00:27.395087 2286 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:27.395251 2286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:27.402172 2286 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 20:00:27.405820 2304 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:27.407670 2286 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:27.407932 2286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "68828397ae4e4c9386f67192df91d950"
format_stamp: "Formatted at 2026-08-12 20:00:22 on dist-test-slave-p3t5"
I20260812 20:00:27.408841 2286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:27.433604 2286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:27.434788 2286 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:27.435146 2286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:27.457883 2286 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:41971
I20260812 20:00:27.457929 2362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:41971 every 8 connection(s)
I20260812 20:00:27.459806 2286 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:00:27.464038 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2286
I20260812 20:00:27.464382 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2112
I20260812 20:00:27.468454 2363 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:27.473174 2363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Bootstrap starting.
I20260812 20:00:27.478094 2255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:27.494323 2363 log.cc:826] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:27.506464 2363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:27.507149 2363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: Bootstrap complete.
I20260812 20:00:27.514961 2363 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } }
I20260812 20:00:27.515935 2363 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68828397ae4e4c9386f67192df91d950, State: Initialized, Role: FOLLOWER
I20260812 20:00:27.516901 2363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } }
I20260812 20:00:27.517930 2369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: true } } }
I20260812 20:00:27.518297 2369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:27.519109 2363 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:27.531090 2383 catalog_manager.cc:1295] Loaded cluster ID: 28a701bb84404b98b31eed303b643527
I20260812 20:00:27.531353 2383 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: loading cluster ID for follower catalog manager: success
I20260812 20:00:27.537081 2383 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: acquiring CA information for follower catalog manager: success
I20260812 20:00:27.540964 2383 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:27.597658 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2112
I20260812 20:00:27.626067 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40889
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--webserver_port=35999
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437 with env {}
I20260812 20:00:27.921164 2335 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee06fae2293f47a89a3a47e7f2f1e650 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:27.929520 2394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:27.930080 2394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:27.930356 2369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176f25e4cfdd41c08e9a3a8afc8f03da. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:27.931875 2369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68828397ae4e4c9386f67192df91d950 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:00:28.050026 2390 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:28.050420 2390 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:28.050565 2390 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:28.059774 2390 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:28.060027 2390 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:28.071749 2390 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35999
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:28.073879 2390 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:28.076086 2390 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:28.087718 2401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:28.088406 2402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:28.088968 2404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:28.089937 2390 server_base.cc:1061] running on GCE node
I20260812 20:00:28.090682 2390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:28.091835 2390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:28.093117 2390 hybrid_clock.cc:648] HybridClock initialized: now 1786564828093064 us; error 82 us; skew 500 ppm
I20260812 20:00:28.093525 2390 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:28.096432 2390 webserver.cc:533] Webserver started at http://127.25.14.1:35999/ using document root <none> and password file <none>
I20260812 20:00:28.097585 2390 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:28.097800 2390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:28.104511 2390 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:00:28.108316 2410 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:28.110599 2390 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:28.110864 2390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:28.111835 2390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:28.137066 2390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:28.138254 2390 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:28.138643 2390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:28.139988 2390 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:28.142324 2390 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:28.142465 2390 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:28.142652 2390 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:28.142740 2390 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:28.185037 2390 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40889
I20260812 20:00:28.185084 2535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40889 every 8 connection(s)
I20260812 20:00:28.187089 2390 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:28.191169 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2390
I20260812 20:00:28.240836 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2390
I20260812 20:00:28.270777 2537 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41971
I20260812 20:00:28.270881 2539 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37727
I20260812 20:00:28.271376 2537 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:28.271384 2539 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:28.270843 2538 heartbeater.cc:344] Connected to a master server at 127.25.14.59:40437
I20260812 20:00:28.271687 2538 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:28.272928 2537 heartbeater.cc:507] Master 127.25.14.62:41971 requested a full tablet report, sending...
I20260812 20:00:28.273488 2538 heartbeater.cc:507] Master 127.25.14.59:40437 requested a full tablet report, sending...
I20260812 20:00:28.274083 2536 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40235
I20260812 20:00:28.274231 2536 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:28.275321 2539 heartbeater.cc:507] Master 127.25.14.61:37727 requested a full tablet report, sending...
I20260812 20:00:28.276208 2053 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:28.276655 2536 heartbeater.cc:507] Master 127.25.14.60:40235 requested a full tablet report, sending...
I20260812 20:00:28.276633 1542 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:28.276638 2324 ts_manager.cc:194] Registered new tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:28.277802 1627 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:28.278919 1627 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:33577
I20260812 20:00:28.282557 2531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:28.380182 2276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "68828397ae4e4c9386f67192df91d950" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 41971 } } peers { permanent_uuid: "8d37053881924479a72dabeeb6eb7f8a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37727 } } peers { permanent_uuid: "176f25e4cfdd41c08e9a3a8afc8f03da" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 40235 } } peers { permanent_uuid: "ee06fae2293f47a89a3a47e7f2f1e650" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 40437 } attrs { promote: false } } }
I20260812 20:00:28.380508 2276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:28.384074 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2390
I20260812 20:00:28.411067 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40889
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--webserver_port=35999
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--fault_before_start=1.0 with env {}
W20260812 20:00:28.818238 2558 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:28.818769 2558 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:28.818928 2558 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:00:28.819027 2558 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:28.828967 2558 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:28.829282 2558 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:28.840651 2558 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35999
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:28.842793 2558 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:28.845067 2558 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:28.857940 2565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:28.857941 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:28.858332 2568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:28.859445 2558 server_base.cc:1061] running on GCE node
I20260812 20:00:28.860302 2558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:28.861444 2558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:28.862727 2558 hybrid_clock.cc:648] HybridClock initialized: now 1786564828862638 us; error 91 us; skew 500 ppm
I20260812 20:00:28.863102 2558 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:28.866031 2558 webserver.cc:533] Webserver started at http://127.25.14.1:35999/ using document root <none> and password file <none>
I20260812 20:00:28.867009 2558 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:28.867216 2558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:28.874513 2558 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 20:00:28.878337 2575 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:28.880165 2558 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:00:28.880401 2558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:28.881491 2558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:28.913786 2558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:28.915046 2558 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:28.915473 2558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:28.916882 2558 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:28.919042 2558 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:28.919188 2558 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:28.919330 2558 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:28.919425 2558 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:00:28.919884 2558 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:00:28.920011 2558 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:00:28.920094 2558 fault_injection.cc:59] injecting fault for kudu (pid 2558): FLAGS_fault_before_start (process will exit)
I20260812 20:00:28.947449 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40889
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235
--webserver_port=35999
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437 with env {}
W20260812 20:00:29.405256 2584 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:29.406059 2584 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:29.406275 2584 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:29.422209 2584 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:29.422600 2584 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:29.440433 2584 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=35999
--tserver_master_addrs=127.25.14.62:41971,127.25.14.61:37727,127.25.14.60:40235,127.25.14.59:40437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:29.443570 2584 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:29.447206 2584 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:29.469972 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:29.468497 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:29.473034 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:29.475287 2584 server_base.cc:1061] running on GCE node
I20260812 20:00:29.476647 2584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:29.478765 2584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:29.482227 2584 hybrid_clock.cc:648] HybridClock initialized: now 1786564829482112 us; error 85 us; skew 500 ppm
I20260812 20:00:29.482812 2584 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:29.487496 2584 webserver.cc:533] Webserver started at http://127.25.14.1:35999/ using document root <none> and password file <none>
I20260812 20:00:29.489022 2584 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:29.489327 2584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:29.502089 2584 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.011s sys 0.000s
I20260812 20:00:29.508651 2603 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:29.512184 2584 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260812 20:00:29.512632 2584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "b64ba6b4a19842e9976e6d8e5aef78e4"
format_stamp: "Formatted at 2026-08-12 20:00:24 on dist-test-slave-p3t5"
I20260812 20:00:29.514394 2584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:29.544605 2584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:29.546437 2584 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:29.547083 2584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:29.549011 2584 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:29.552793 2584 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:29.553016 2584 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:29.553267 2584 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:29.553400 2584 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:29.621368 2584 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40889
I20260812 20:00:29.621420 2732 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40889 every 8 connection(s)
I20260812 20:00:29.624511 2584 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:29.626883 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 2584
I20260812 20:00:29.627509 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2584
I20260812 20:00:29.678326 2734 heartbeater.cc:344] Connected to a master server at 127.25.14.62:41971
I20260812 20:00:29.679207 2734 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:29.684770 2737 heartbeater.cc:344] Connected to a master server at 127.25.14.59:40437
I20260812 20:00:29.685360 2737 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:29.686411 2734 heartbeater.cc:507] Master 127.25.14.62:41971 requested a full tablet report, sending...
I20260812 20:00:29.686959 2737 heartbeater.cc:507] Master 127.25.14.59:40437 requested a full tablet report, sending...
I20260812 20:00:29.690651 2053 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:29.692781 2324 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:29.699869 2727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:29.846354 2733 heartbeater.cc:344] Connected to a master server at 127.25.14.60:40235
I20260812 20:00:29.846936 2733 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:29.848059 2733 heartbeater.cc:507] Master 127.25.14.60:40235 requested a full tablet report, sending...
I20260812 20:00:29.850556 1627 ts_manager.cc:194] Re-registered known tserver with Master: b64ba6b4a19842e9976e6d8e5aef78e4 (127.25.14.1:40889)
I20260812 20:00:29.852730 1627 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:45807
I20260812 20:00:29.852847 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2584
W20260812 20:00:29.898789 1608 connection.cc:573] server connection from 127.25.14.1:45807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:29.899495 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 2286
I20260812 20:00:29.910790 2357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:30.089495 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 2286
I20260812 20:00:30.147068 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1504
I20260812 20:00:30.149252 1579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:30.326149 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1504
W20260812 20:00:30.364943 1612 consensus_peers.cc:597] T 00000000000000000000000000000000 P 176f25e4cfdd41c08e9a3a8afc8f03da -> Peer 68828397ae4e4c9386f67192df91d950 (127.25.14.62:41971): Couldn't send request to peer 68828397ae4e4c9386f67192df91d950. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:41971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:30.378247 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1587
I20260812 20:00:30.380520 1660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:30.542883 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1587
W20260812 20:00:30.596264 2037 connection.cc:573] server connection from 127.0.0.1:47128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:30.596976 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 1890
I20260812 20:00:30.599186 2087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:30.764279 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 1890
Shutting down hive metastore.
2026-08-12T20:00:31Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/4 (25846 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/5
2026-08-12T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:31Z Disabled control of system clock
2026-08-12T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:31Z Disabled control of system clock
2026-08-12T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:31Z Disabled control of system clock
2026-08-12T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:31Z Disabled control of system clock
W20260812 20:00:31.695576 25656 mini_chronyd.cc:196] Time spent starting chronyd: real 0.107s user 0.000s sys 0.055s
2026-08-12T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:00:31Z Disabled control of system clock
W20260812 20:00:31.801301 25656 mini_chronyd.cc:196] Time spent starting chronyd: real 0.105s user 0.000s sys 0.053s
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/security.properties
2026-08-12 20:00:39: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260812 20:00:48.664947 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35047
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285 with env {}
W20260812 20:00:49.057545 3100 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:49.058020 3100 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:49.058162 3100 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:49.068066 3100 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:49.068213 3100 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:49.068348 3100 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:49.068432 3100 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:49.079198 3100 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:49.081082 3100 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:49.083279 3100 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:49.092742 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:49.092811 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:49.093418 3110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:49.093796 3100 server_base.cc:1061] running on GCE node
I20260812 20:00:49.094748 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:49.096525 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:49.097795 3100 hybrid_clock.cc:648] HybridClock initialized: now 1786564849097652 us; error 110 us; skew 500 ppm
I20260812 20:00:49.098181 3100 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:49.100526 3100 webserver.cc:533] Webserver started at http://127.25.14.62:34971/ using document root <none> and password file <none>
I20260812 20:00:49.101342 3100 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:49.101488 3100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:49.101933 3100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:49.104383 3100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "387e7e0e7785419b968ff07bee35a4f5"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.105221 3100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "387e7e0e7785419b968ff07bee35a4f5"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.110903 3100 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 20:00:49.114473 3117 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:49.115984 3100 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:00:49.116187 3100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "387e7e0e7785419b968ff07bee35a4f5"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.116442 3100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:49.134135 3100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:49.135123 3100 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:49.135447 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:49.157310 3100 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35047
I20260812 20:00:49.157330 3170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35047 every 8 connection(s)
I20260812 20:00:49.159081 3100 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:00:49.164093 3171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:49.166297 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3100
I20260812 20:00:49.166519 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:00:49.168087 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46621
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285 with env {}
I20260812 20:00:49.173415 3171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:49.192919 3171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:49.194521 3120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
W20260812 20:00:49.196727 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46621: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
I20260812 20:00:49.246170 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } attempt: 1
W20260812 20:00:49.248502 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46621: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
I20260812 20:00:49.316847 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } attempt: 2
W20260812 20:00:49.319373 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46621: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
I20260812 20:00:49.410705 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } attempt: 3
W20260812 20:00:49.413093 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46621: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
I20260812 20:00:49.556370 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } attempt: 4
W20260812 20:00:49.559096 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:46621: Network error: Client connection negotiation failed: client connection to 127.25.14.61:46621: connect: Connection refused (error 111)
W20260812 20:00:49.569839 3175 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:49.570297 3175 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:49.570436 3175 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:49.579452 3175 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:49.579614 3175 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:49.579741 3175 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:49.579823 3175 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:49.590946 3175 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:49.592844 3175 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:49.595029 3175 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:49.604456 3184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:49.604435 3183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:49.605022 3187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:49.607267 3175 server_base.cc:1061] running on GCE node
I20260812 20:00:49.608268 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:49.610260 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:49.611626 3175 hybrid_clock.cc:648] HybridClock initialized: now 1786564849611465 us; error 118 us; skew 500 ppm
I20260812 20:00:49.612109 3175 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:49.614809 3175 webserver.cc:533] Webserver started at http://127.25.14.61:40849/ using document root <none> and password file <none>
I20260812 20:00:49.615722 3175 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:49.615873 3175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:49.616274 3175 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:49.619204 3175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "935d52cbf38e4bf785f5e7158c65402a"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.620069 3175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "935d52cbf38e4bf785f5e7158c65402a"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.626338 3175 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260812 20:00:49.630309 3195 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:49.632180 3175 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:49.632424 3175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "935d52cbf38e4bf785f5e7158c65402a"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:49.632697 3175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:49.693559 3175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:49.694622 3175 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:49.694959 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:49.717898 3175 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:46621
I20260812 20:00:49.717916 3254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:46621 every 8 connection(s)
I20260812 20:00:49.719712 3175 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:00:49.723032 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3175
I20260812 20:00:49.723284 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:00:49.724385 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:33285
--webserver_interface=127.25.14.60
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285 with env {}
I20260812 20:00:49.725258 3256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:49.731935 3256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:49.746655 3256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:49.757529 3256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:49.758939 3198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
W20260812 20:00:49.760865 3256 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.810253 3256 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 1
W20260812 20:00:49.812743 3256 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.858472 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } attempt: 5
I20260812 20:00:49.866088 3171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:00:49.868258 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.881026 3256 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 2
W20260812 20:00:49.883692 3256 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.919603 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 1
W20260812 20:00:49.922556 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.974997 3256 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 3
W20260812 20:00:49.977548 3256 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:49.990808 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 2
W20260812 20:00:49.993149 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:50.099449 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 3
W20260812 20:00:50.101820 3171 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
I20260812 20:00:50.120870 3256 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 4
W20260812 20:00:50.123569 3256 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:33285: Network error: Client connection negotiation failed: client connection to 127.25.14.60:33285: connect: Connection refused (error 111)
W20260812 20:00:50.135277 3261 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:50.135712 3261 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:50.135860 3261 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:50.144825 3261 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:50.144995 3261 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:50.145123 3261 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:50.145210 3261 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:50.156136 3261 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:50.158082 3261 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:50.160109 3261 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:50.170137 3268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:50.170234 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:50.170456 3271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:50.171213 3261 server_base.cc:1061] running on GCE node
I20260812 20:00:50.172130 3261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:50.174000 3261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:50.175251 3261 hybrid_clock.cc:648] HybridClock initialized: now 1786564850175079 us; error 138 us; skew 500 ppm
I20260812 20:00:50.175638 3261 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:50.178073 3261 webserver.cc:533] Webserver started at http://127.25.14.60:35003/ using document root <none> and password file <none>
I20260812 20:00:50.178894 3261 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:50.179044 3261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:50.179464 3261 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:50.182232 3261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5"
format_stamp: "Formatted at 2026-08-12 20:00:50 on dist-test-slave-p3t5"
I20260812 20:00:50.183138 3261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5"
format_stamp: "Formatted at 2026-08-12 20:00:50 on dist-test-slave-p3t5"
I20260812 20:00:50.189153 3261 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:00:50.193066 3278 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:50.194749 3261 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:00:50.194998 3261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5"
format_stamp: "Formatted at 2026-08-12 20:00:50 on dist-test-slave-p3t5"
I20260812 20:00:50.195254 3261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:50.220350 3261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:50.221413 3261 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:50.221772 3261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:50.244923 3334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:33285 every 8 connection(s)
I20260812 20:00:50.244962 3261 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:33285
I20260812 20:00:50.246790 3261 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:00:50.250180 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3261
I20260812 20:00:50.250417 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:00:50.252382 3336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:50.260787 3336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:50.276261 3336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:50.279083 3171 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 4
I20260812 20:00:50.284698 3336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:50.296247 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Bootstrap starting.
I20260812 20:00:50.300873 3171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:50.302495 3171 log.cc:826] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:50.302623 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Bootstrap starting.
I20260812 20:00:50.306731 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: No bootstrap required, opened a new log
I20260812 20:00:50.307013 3336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:50.308477 3336 log.cc:826] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:50.311702 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: No bootstrap required, opened a new log
I20260812 20:00:50.314627 3171 raft_consensus.cc:359] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.315164 3171 raft_consensus.cc:385] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:50.315310 3171 raft_consensus.cc:740] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387e7e0e7785419b968ff07bee35a4f5, State: Initialized, Role: FOLLOWER
I20260812 20:00:50.316180 3171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.317232 3347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.317533 3347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.318706 3171 sys_catalog.cc:565] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:50.319440 3336 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.319832 3336 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:50.319974 3336 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf9a0ccea15427f90dc4ba4824e3bc5, State: Initialized, Role: FOLLOWER
I20260812 20:00:50.320863 3336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.322217 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.322618 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.323710 3336 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:50.331943 3100 master_runner.cc:187] Error getting master registration for 127.25.14.61:46621: OK, instance_id { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" instance_seqno: 1786564849711701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:00:50.339083 3100 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 20:00:50.341336 3361 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:50.341720 3361 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 20:00:50.347054 3374 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:50.347328 3374 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:50.344589 3261 master_runner.cc:187] Error getting master registration for 127.25.14.61:46621: OK, instance_id { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" instance_seqno: 1786564849711701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:00:50.350214 3261 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 20:00:50.388608 3123 tablet.cc:2406] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:00:50.422869 3256 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } attempt: 5
I20260812 20:00:50.433944 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: Bootstrap starting.
I20260812 20:00:50.439388 3256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:50.440618 3256 log.cc:826] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:50.443542 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: No bootstrap required, opened a new log
I20260812 20:00:50.449838 3256 raft_consensus.cc:359] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.450181 3256 raft_consensus.cc:385] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:50.450313 3256 raft_consensus.cc:740] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935d52cbf38e4bf785f5e7158c65402a, State: Initialized, Role: FOLLOWER
I20260812 20:00:50.451061 3256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.452183 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.452558 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.453567 3256 sys_catalog.cc:565] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:00:50.474268 3284 tablet.cc:2406] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:00:50.477789 3392 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:50.478031 3392 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:50.481177 3175 master_runner.cc:264] No leader master found from master 935d52cbf38e4bf785f5e7158c65402a
I20260812 20:00:50.481317 3175 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:00:50.530126 3354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:00:50.530508 3354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.532616 3354 leader_election.cc:290] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 387e7e0e7785419b968ff07bee35a4f5 (127.25.14.62:35047), 935d52cbf38e4bf785f5e7158c65402a (127.25.14.61:46621)
I20260812 20:00:50.534193 3227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d52cbf38e4bf785f5e7158c65402a" is_pre_election: true
I20260812 20:00:50.534847 3227 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf9a0ccea15427f90dc4ba4824e3bc5 in term 0.
I20260812 20:00:50.536327 3281 leader_election.cc:304] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf9a0ccea15427f90dc4ba4824e3bc5, 935d52cbf38e4bf785f5e7158c65402a; no voters:
I20260812 20:00:50.537338 3144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387e7e0e7785419b968ff07bee35a4f5" is_pre_election: true
I20260812 20:00:50.537946 3144 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf9a0ccea15427f90dc4ba4824e3bc5 in term 0.
I20260812 20:00:50.539381 3354 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:00:50.539558 3354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:00:50.539669 3354 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:50.543452 3354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.544909 3144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387e7e0e7785419b968ff07bee35a4f5"
I20260812 20:00:50.545153 3144 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:50.548974 3144 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf9a0ccea15427f90dc4ba4824e3bc5 in term 1.
I20260812 20:00:50.549728 3282 leader_election.cc:304] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf9a0ccea15427f90dc4ba4824e3bc5, 387e7e0e7785419b968ff07bee35a4f5; no voters:
I20260812 20:00:50.558651 3227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d52cbf38e4bf785f5e7158c65402a"
I20260812 20:00:50.558934 3227 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:00:50.563074 3227 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf9a0ccea15427f90dc4ba4824e3bc5 in term 1.
I20260812 20:00:50.564308 3354 leader_election.cc:290] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [CANDIDATE]: Term 1 election: Requested vote from peers 387e7e0e7785419b968ff07bee35a4f5 (127.25.14.62:35047), 935d52cbf38e4bf785f5e7158c65402a (127.25.14.61:46621)
I20260812 20:00:50.564937 3354 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:00:50.565591 3354 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf9a0ccea15427f90dc4ba4824e3bc5, State: Running, Role: LEADER
I20260812 20:00:50.566696 3354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } }
I20260812 20:00:50.581548 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf9a0ccea15427f90dc4ba4824e3bc5. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.582137 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:50.589583 3396 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:00:50.592326 3396 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:00:50.614674 3227 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:00:50.616006 3394 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:00:50.624107 3144 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:00:50.625370 3394 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:00:50.651012 3347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf9a0ccea15427f90dc4ba4824e3bc5. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.651391 3347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.657806 3400 mvcc.cc:204] Tried to move back new op lower bound from 7317769628102406144 to 7317769627938017280. Current Snapshot: MvccSnapshot[applied={T|T < 7317769628102406144}]
I20260812 20:00:50.658577 3398 mvcc.cc:204] Tried to move back new op lower bound from 7317769628102406144 to 7317769627938017280. Current Snapshot: MvccSnapshot[applied={T|T < 7317769628102406144}]
I20260812 20:00:50.671110 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.671463 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:50.672367 3347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.672624 3347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.673045 3396 catalog_manager.cc:1383] Generated new cluster ID: 138086123b1e497c9f942593a75cb25d
I20260812 20:00:50.673166 3396 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
W20260812 20:00:50.697949 3201 tablet.cc:2406] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:00:50.701390 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf9a0ccea15427f90dc4ba4824e3bc5. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.701700 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.705964 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.706238 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:50.709174 3399 mvcc.cc:204] Tried to move back new op lower bound from 7317769628102406144 to 7317769627938017280. Current Snapshot: MvccSnapshot[applied={T|T < 7317769628102406144}]
I20260812 20:00:50.734658 3396 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:00:50.737272 3396 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:00:50.743311 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } }
I20260812 20:00:50.743582 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:50.811685 3396 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Generated new TSK 0
I20260812 20:00:50.813061 3396 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 20:00:50.847628 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260812 20:00:51.302583 3407 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:51.303007 3407 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:51.303138 3407 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:51.311983 3407 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:51.312220 3407 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:51.323149 3407 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:51.325125 3407 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:51.327276 3407 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:51.338413 3412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:51.338649 3415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:51.338689 3413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:51.339967 3407 server_base.cc:1061] running on GCE node
I20260812 20:00:51.340622 3407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:51.341969 3407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:51.344125 3361 catalog_manager.cc:1295] Loaded cluster ID: 138086123b1e497c9f942593a75cb25d
I20260812 20:00:51.344213 3361 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: loading cluster ID for follower catalog manager: success
I20260812 20:00:51.345360 3407 hybrid_clock.cc:648] HybridClock initialized: now 1786564851345165 us; error 154 us; skew 500 ppm
I20260812 20:00:51.345798 3407 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:51.347306 3361 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: acquiring CA information for follower catalog manager: success
I20260812 20:00:51.348443 3407 webserver.cc:533] Webserver started at http://127.25.14.1:37475/ using document root <none> and password file <none>
I20260812 20:00:51.349341 3407 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:51.349498 3407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:51.349690 3361 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:51.350072 3407 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:51.353195 3407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.354131 3407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.360627 3407 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:00:51.365226 3424 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:51.367092 3407 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:51.367347 3407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.367631 3407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:51.389607 3407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:51.390686 3407 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:51.391048 3407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:51.392143 3407 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:51.394248 3407 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:51.394384 3407 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:51.394524 3407 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:51.394615 3407 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:51.430809 3407 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46837
I20260812 20:00:51.430950 3552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46837 every 8 connection(s)
I20260812 20:00:51.432400 3407 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:51.434049 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3407
I20260812 20:00:51.434307 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 20:00:51.470474 3555 heartbeater.cc:344] Connected to a master server at 127.25.14.60:33285
I20260812 20:00:51.471016 3555 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:51.471156 3556 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46621
I20260812 20:00:51.471311 3556 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:51.471863 3554 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35047
I20260812 20:00:51.472040 3554 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:51.472478 3556 heartbeater.cc:507] Master 127.25.14.61:46621 requested a full tablet report, sending...
I20260812 20:00:51.472510 3555 heartbeater.cc:507] Master 127.25.14.60:33285 requested a full tablet report, sending...
I20260812 20:00:51.472608 3554 heartbeater.cc:507] Master 127.25.14.62:35047 requested a full tablet report, sending...
I20260812 20:00:51.475023 3216 ts_manager.cc:194] Registered new tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:51.475106 3295 ts_manager.cc:194] Registered new tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:51.475124 3133 ts_manager.cc:194] Registered new tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:51.476841 3295 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:32871
I20260812 20:00:51.479984 3392 catalog_manager.cc:1295] Loaded cluster ID: 138086123b1e497c9f942593a75cb25d
I20260812 20:00:51.480111 3392 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: loading cluster ID for follower catalog manager: success
I20260812 20:00:51.483564 3392 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: acquiring CA information for follower catalog manager: success
I20260812 20:00:51.486375 3392 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:51.487057 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:51.489506 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:38295
--webserver_interface=127.25.14.59
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295 with env {}
W20260812 20:00:51.882247 3566 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:51.882647 3566 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:51.882727 3566 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:51.891678 3566 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:51.891800 3566 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:51.891908 3566 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:51.892009 3566 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:51.902894 3566 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:51.904742 3566 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:51.906843 3566 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:51.915906 3573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:51.915936 3574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:51.916673 3576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:51.917045 3566 server_base.cc:1061] running on GCE node
I20260812 20:00:51.918011 3566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:51.920046 3566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:51.921442 3566 hybrid_clock.cc:648] HybridClock initialized: now 1786564851921241 us; error 146 us; skew 500 ppm
I20260812 20:00:51.921898 3566 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:51.924566 3566 webserver.cc:533] Webserver started at http://127.25.14.59:35317/ using document root <none> and password file <none>
I20260812 20:00:51.925522 3566 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:51.925719 3566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:51.926179 3566 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:00:51.928853 3566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.929876 3566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.936069 3566 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:00:51.939970 3583 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:51.941637 3566 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:00:51.941869 3566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:51.942164 3566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:51.980109 3566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:51.981195 3566 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:51.981501 3566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:52.003867 3642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:38295 every 8 connection(s)
I20260812 20:00:52.003877 3566 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:38295
I20260812 20:00:52.005963 3566 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:52.011379 3643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:00:52.011873 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3566
I20260812 20:00:52.020030 3643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:52.031728 3643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:52.039089 3643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:52.046278 3643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:00:52.057824 3643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Bootstrap starting.
I20260812 20:00:52.061455 3643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Neither blocks nor log segments found. Creating new log.
I20260812 20:00:52.062783 3643 log.cc:826] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:52.065740 3643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: No bootstrap required, opened a new log
I20260812 20:00:52.071228 3643 raft_consensus.cc:359] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } }
I20260812 20:00:52.071560 3643 raft_consensus.cc:385] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:00:52.071676 3643 raft_consensus.cc:740] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a3efdeb9524b2297076e2a9a34d4fd, State: Initialized, Role: FOLLOWER
I20260812 20:00:52.072412 3643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } }
I20260812 20:00:52.073469 3655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } } }
I20260812 20:00:52.073861 3655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:52.074760 3643 sys_catalog.cc:565] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:00:52.088424 3666 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:00:52.088682 3666 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:00:52.096586 3566 master_runner.cc:429] Detected that this master 08a3efdeb9524b2297076e2a9a34d4fd is joining an existing cluster
I20260812 20:00:52.096724 3566 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:00:52.097820 3566 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:00:52.098205 3566 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:00:52.098469 3566 tablet_replica.cc:333] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: stopping tablet replica
I20260812 20:00:52.100154 3566 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:38295
I20260812 20:00:52.141335 3295 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:38295: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 38295
}
attrs {
promote: true
}
}
dest_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5"
cas_config_opid_index: -1
I20260812 20:00:52.142932 3295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } }
I20260812 20:00:52.146163 3143 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:00:52.146971 3227 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:00:52.147053 3354 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260812 20:00:52.148169 3279 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 -> Peer 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59:38295): Couldn't send request to peer 08a3efdeb9524b2297076e2a9a34d4fd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:00:52.148197 3354 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:00:52.151572 3680 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.152321 3143 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.152959 3227 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.156836 3680 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:38295
I20260812 20:00:52.168191 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.168582 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:52.168735 3566 master.cc:562] Master@127.25.14.59:38295 shutting down...
I20260812 20:00:52.179913 3687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.168874 3685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.181648 3687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:52.181388 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.181871 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:52.182565 3685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:52.183367 3688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.183679 3688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:00:52.214524 3587 connection.cc:466] server connection from 127.0.0.1:44260 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:44260 (request call id 11) could send its response
I20260812 20:00:52.227563 3566 master.cc:584] Master@127.25.14.59:38295 shutdown complete.
I20260812 20:00:52.227681 3566 master_runner.cc:306] Clearing existing system tablet
I20260812 20:00:52.228144 3566 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:52.228272 3566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:52.231379 3566 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260812 20:00:52.233183 3692 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:52.233721 3566 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:52.234018 3566 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:00:52.234128 3566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:52.239127 3566 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:00:52.246915 3566 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:00:52.247143 3566 log.cc:1199] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:00:52.247732 3566 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Deleting consensus metadata
I20260812 20:00:52.248035 3566 master_runner.cc:316] Copying system tablet from 127.25.14.60:33285
I20260812 20:00:52.251257 3566 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:33285
I20260812 20:00:52.261307 3318 tablet_copy_service.cc:140] P 1cf9a0ccea15427f90dc4ba4824e3bc5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08a3efdeb9524b2297076e2a9a34d4fd ({username='slave'} at 127.0.0.1:56596)
I20260812 20:00:52.261662 3318 tablet_copy_service.cc:161] P 1cf9a0ccea15427f90dc4ba4824e3bc5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08a3efdeb9524b2297076e2a9a34d4fd at {username='slave'} at 127.0.0.1:56596: session id = 08a3efdeb9524b2297076e2a9a34d4fd-00000000000000000000000000000000
I20260812 20:00:52.264811 3318 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:00:52.267472 3566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:00:52.274842 3566 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: tablet copy: Starting download of 0 data blocks...
I20260812 20:00:52.275187 3566 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: tablet copy: Starting download of 1 WAL segments...
I20260812 20:00:52.278348 3566 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:00:52.283507 3318 tablet_copy_service.cc:342] P 1cf9a0ccea15427f90dc4ba4824e3bc5: Request end of tablet copy session 08a3efdeb9524b2297076e2a9a34d4fd-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:56596
I20260812 20:00:52.283774 3318 tablet_copy_service.cc:434] P 1cf9a0ccea15427f90dc4ba4824e3bc5: ending tablet copy session 08a3efdeb9524b2297076e2a9a34d4fd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08a3efdeb9524b2297076e2a9a34d4fd
W20260812 20:00:52.291848 3566 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:00:52.295641 3566 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:52.298504 3701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:52.299873 3700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:52.305523 3703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:52.306422 3566 server_base.cc:1061] running on GCE node
I20260812 20:00:52.306829 3566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:52.307672 3566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:52.308913 3566 hybrid_clock.cc:648] HybridClock initialized: now 1786564852308782 us; error 85 us; skew 500 ppm
I20260812 20:00:52.309160 3566 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:52.310734 3566 webserver.cc:533] Webserver started at http://127.25.14.59:39017/ using document root <none> and password file <none>
I20260812 20:00:52.311111 3566 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:52.311219 3566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:52.313853 3566 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 20:00:52.315867 3711 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:52.316416 3566 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:00:52.316570 3566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "08a3efdeb9524b2297076e2a9a34d4fd"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:52.316747 3566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:52.327155 3566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:52.327953 3566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:52.348172 3566 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:38295
I20260812 20:00:52.348258 3769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:38295 every 8 connection(s)
I20260812 20:00:52.349330 3566 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:00:52.353282 3770 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:52.355912 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Bootstrap starting.
I20260812 20:00:52.373903 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:52.374567 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: Bootstrap complete.
I20260812 20:00:52.375998 3770 raft_consensus.cc:359] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } }
I20260812 20:00:52.376317 3770 raft_consensus.cc:740] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08a3efdeb9524b2297076e2a9a34d4fd, State: Initialized, Role: LEARNER
I20260812 20:00:52.376725 3770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } }
I20260812 20:00:52.377874 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.378237 3770 sys_catalog.cc:565] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:52.378275 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: This master's current role is: LEARNER
I20260812 20:00:52.387640 3790 catalog_manager.cc:1295] Loaded cluster ID: 138086123b1e497c9f942593a75cb25d
I20260812 20:00:52.387748 3790 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: loading cluster ID for follower catalog manager: success
I20260812 20:00:52.390687 3790 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: acquiring CA information for follower catalog manager: success
I20260812 20:00:52.392861 3790 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:52.430987 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3407
I20260812 20:00:52.442492 3548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:52.550127 3555 heartbeater.cc:499] Master 127.25.14.60:33285 was elected leader, sending a full tablet report...
I20260812 20:00:52.552698 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3407
I20260812 20:00:52.582913 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46837
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--webserver_port=37475
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295 with env {}
I20260812 20:00:52.734861 3744 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:00:52.735740 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf9a0ccea15427f90dc4ba4824e3bc5. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:52.735991 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: This master's current role is: LEARNER
W20260812 20:00:52.985085 3793 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:52.985437 3793 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:52.985530 3793 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:52.994259 3793 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:52.994443 3793 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:53.004907 3793 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46837
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37475
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:53.006768 3793 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:53.008723 3793 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:53.019932 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:53.019817 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:53.020162 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:53.020998 3793 server_base.cc:1061] running on GCE node
I20260812 20:00:53.021742 3793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:53.023074 3793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:53.024284 3793 hybrid_clock.cc:648] HybridClock initialized: now 1786564853024142 us; error 110 us; skew 500 ppm
I20260812 20:00:53.024647 3793 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:53.027052 3793 webserver.cc:533] Webserver started at http://127.25.14.1:37475/ using document root <none> and password file <none>
I20260812 20:00:53.027889 3793 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:53.028050 3793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:53.034420 3793 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:00:53.037734 3809 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:53.039621 3793 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:53.039893 3793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:53.040716 3793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:53.065459 3793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:53.066433 3793 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:53.066690 3793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:53.067739 3793 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:53.069566 3793 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:53.069700 3793 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 20:00:53.069818 3793 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:53.069868 3793 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:53.110026 3793 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46837
I20260812 20:00:53.110110 3944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46837 every 8 connection(s)
I20260812 20:00:53.111733 3793 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:53.116385 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3793
I20260812 20:00:53.150764 3948 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46621
I20260812 20:00:53.151134 3946 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35047
I20260812 20:00:53.151320 3946 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:53.151301 3948 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:53.151494 3947 heartbeater.cc:344] Connected to a master server at 127.25.14.60:33285
I20260812 20:00:53.151650 3947 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:53.151291 3945 heartbeater.cc:344] Connected to a master server at 127.25.14.59:38295
I20260812 20:00:53.151949 3945 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:53.152741 3948 heartbeater.cc:507] Master 127.25.14.61:46621 requested a full tablet report, sending...
I20260812 20:00:53.152782 3946 heartbeater.cc:507] Master 127.25.14.62:35047 requested a full tablet report, sending...
I20260812 20:00:53.152841 3945 heartbeater.cc:507] Master 127.25.14.59:38295 requested a full tablet report, sending...
I20260812 20:00:53.153381 3947 heartbeater.cc:507] Master 127.25.14.60:33285 requested a full tablet report, sending...
I20260812 20:00:53.155306 3216 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:53.155421 3133 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:53.155517 3731 ts_manager.cc:194] Registered new tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:53.156255 3295 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:53.157385 3295 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:36705
I20260812 20:00:53.160163 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:53.169471 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:00:53.169736 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3100
I20260812 20:00:53.170630 3166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:53.241873 3680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:53.242261 3680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:53.243933 3680 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5: attempting to promote NON_VOTER 08a3efdeb9524b2297076e2a9a34d4fd to VOTER
I20260812 20:00:53.245095 3680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } }
I20260812 20:00:53.248220 3227 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:53.248302 3744 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 1 LEARNER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:53.249006 3680 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:00:53.249341 3959 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:00:53.259964 3959 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.263751 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.264151 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:53.264465 3227 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.264796 3964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.265230 3964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:53.265167 3744 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.269073 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.269294 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:53.269955 3965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.270431 3965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a3efdeb9524b2297076e2a9a34d4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:53.269744 3966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.270808 3966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d52cbf38e4bf785f5e7158c65402a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:53.293347 3143 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Refusing update from remote peer 1cf9a0ccea15427f90dc4ba4824e3bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:00:53.294188 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3100
I20260812 20:00:53.294270 3958 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 20:00:53.323338 3282 connection.cc:573] client connection to 127.25.14.62:35047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 20:00:53.323683 3282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:00:53.324431 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35047
--webserver_interface=127.25.14.62
--webserver_port=34971
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295 with env {}
W20260812 20:00:53.326301 3282 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 -> Peer 387e7e0e7785419b968ff07bee35a4f5 (127.25.14.62:35047): Couldn't send request to peer 387e7e0e7785419b968ff07bee35a4f5. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:00:53.717100 3970 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:53.717461 3970 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:53.717639 3970 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:53.731741 3970 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:00:53.731886 3970 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:53.731966 3970 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:00:53.732018 3970 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:00:53.744407 3970 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=34971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:53.746169 3970 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:53.748214 3970 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:53.757170 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:53.757454 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:53.757661 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:53.758836 3970 server_base.cc:1061] running on GCE node
I20260812 20:00:53.759733 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:53.761541 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:53.762988 3970 hybrid_clock.cc:648] HybridClock initialized: now 1786564853762814 us; error 135 us; skew 500 ppm
I20260812 20:00:53.763399 3970 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:53.766088 3970 webserver.cc:533] Webserver started at http://127.25.14.62:34971/ using document root <none> and password file <none>
I20260812 20:00:53.766944 3970 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:53.767107 3970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:53.773161 3970 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 20:00:53.776692 3987 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:53.778620 3970 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:53.778858 3970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "387e7e0e7785419b968ff07bee35a4f5"
format_stamp: "Formatted at 2026-08-12 20:00:49 on dist-test-slave-p3t5"
I20260812 20:00:53.779738 3970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:53.805881 3970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:53.807009 3970 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:53.807353 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:53.829677 3970 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35047
I20260812 20:00:53.829706 4053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35047 every 8 connection(s)
I20260812 20:00:53.831627 3970 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:00:53.832556 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 3970
I20260812 20:00:53.832917 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3793
I20260812 20:00:53.840725 4054 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:00:53.842921 3940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:53.845880 4054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Bootstrap starting.
I20260812 20:00:53.871869 4054 log.cc:826] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Log is configured to *not* fsync() on all Append() calls
I20260812 20:00:53.887542 4054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:00:53.888363 4054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: Bootstrap complete.
I20260812 20:00:53.897291 4054 raft_consensus.cc:359] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } }
I20260812 20:00:53.898430 4054 raft_consensus.cc:740] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387e7e0e7785419b968ff07bee35a4f5, State: Initialized, Role: FOLLOWER
I20260812 20:00:53.899296 4054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } }
I20260812 20:00:53.900314 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: true } } }
I20260812 20:00:53.900687 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:53.901923 4054 sys_catalog.cc:565] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:00:53.910953 4022 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08a3efdeb9524b2297076e2a9a34d4fd (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.918459 4076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.919015 4076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:53.919615 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf9a0ccea15427f90dc4ba4824e3bc5. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:53.920172 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:00:53.927304 4077 catalog_manager.cc:1295] Loaded cluster ID: 138086123b1e497c9f942593a75cb25d
I20260812 20:00:53.927497 4077 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: loading cluster ID for follower catalog manager: success
I20260812 20:00:53.932039 4077 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: acquiring CA information for follower catalog manager: success
I20260812 20:00:53.934826 4077 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 387e7e0e7785419b968ff07bee35a4f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:00:53.971592 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3793
I20260812 20:00:54.005249 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46837
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--webserver_port=37475
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295 with env {}
W20260812 20:00:54.416672 4083 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:54.417076 4083 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:54.417163 4083 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:54.427572 4083 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:54.427771 4083 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:54.439685 4083 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46837
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37475
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:54.441447 4083 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:54.443568 4083 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:54.455430 4091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:54.455430 4089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:54.455760 4088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:54.456694 4083 server_base.cc:1061] running on GCE node
I20260812 20:00:54.457719 4083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:54.458377 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "387e7e0e7785419b968ff07bee35a4f5" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35047 } } peers { permanent_uuid: "935d52cbf38e4bf785f5e7158c65402a" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 46621 } } peers { permanent_uuid: "1cf9a0ccea15427f90dc4ba4824e3bc5" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 33285 } } peers { permanent_uuid: "08a3efdeb9524b2297076e2a9a34d4fd" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38295 } attrs { promote: false } } }
I20260812 20:00:54.458654 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf9a0ccea15427f90dc4ba4824e3bc5 [sys.catalog]: This master's current role is: LEADER
I20260812 20:00:54.459425 4083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:54.462823 4083 hybrid_clock.cc:648] HybridClock initialized: now 1786564854462571 us; error 201 us; skew 500 ppm
I20260812 20:00:54.463306 4083 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:54.466107 4083 webserver.cc:533] Webserver started at http://127.25.14.1:37475/ using document root <none> and password file <none>
I20260812 20:00:54.467056 4083 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:54.467207 4083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:54.473383 4083 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 20:00:54.476802 4101 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:54.478577 4083 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:54.478817 4083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:54.479596 4083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:54.531286 4083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:54.532322 4083 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:54.532708 4083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:54.533923 4083 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:54.535897 4083 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:54.536018 4083 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:54.536177 4083 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:54.536253 4083 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:54.582544 4083 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46837
I20260812 20:00:54.582584 4232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46837 every 8 connection(s)
I20260812 20:00:54.584290 4083 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:54.590927 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4083
I20260812 20:00:54.631187 4233 heartbeater.cc:344] Connected to a master server at 127.25.14.59:38295
I20260812 20:00:54.631683 4233 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:54.633008 4233 heartbeater.cc:507] Master 127.25.14.59:38295 requested a full tablet report, sending...
I20260812 20:00:54.634343 4236 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46621
I20260812 20:00:54.634495 4236 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:54.635036 4236 heartbeater.cc:507] Master 127.25.14.61:46621 requested a full tablet report, sending...
I20260812 20:00:54.636500 3216 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:54.636718 3731 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:54.641520 4234 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35047
I20260812 20:00:54.641773 4234 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:54.642462 4234 heartbeater.cc:507] Master 127.25.14.62:35047 requested a full tablet report, sending...
I20260812 20:00:54.644747 4009 ts_manager.cc:194] Registered new tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:54.649214 4235 heartbeater.cc:344] Connected to a master server at 127.25.14.60:33285
I20260812 20:00:54.649451 4235 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:54.650097 4235 heartbeater.cc:507] Master 127.25.14.60:33285 requested a full tablet report, sending...
I20260812 20:00:54.651593 3295 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:54.652840 3295 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:43107
I20260812 20:00:54.657858 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4083
I20260812 20:00:54.670255 4228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:54.775775 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4083
I20260812 20:00:54.802281 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46837
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--webserver_port=37475
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--fault_before_start=1.0 with env {}
W20260812 20:00:55.204257 4253 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:55.204641 4253 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:55.204824 4253 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:00:55.204921 4253 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:55.214236 4253 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:55.214449 4253 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:55.225176 4253 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46837
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37475
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:55.227103 4253 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:55.228976 4253 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:55.239513 4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:55.240140 4259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:55.240424 4261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:55.241531 4253 server_base.cc:1061] running on GCE node
I20260812 20:00:55.242305 4253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:55.243659 4253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:55.245046 4253 hybrid_clock.cc:648] HybridClock initialized: now 1786564855244848 us; error 126 us; skew 500 ppm
I20260812 20:00:55.245411 4253 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:55.247984 4253 webserver.cc:533] Webserver started at http://127.25.14.1:37475/ using document root <none> and password file <none>
I20260812 20:00:55.248808 4253 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:55.248971 4253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:55.255722 4253 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:55.259048 4270 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:55.260967 4253 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:55.261214 4253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:55.262187 4253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:55.289124 4253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:55.290385 4253 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:55.290767 4253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:55.292034 4253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:55.294298 4253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:55.294415 4253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:55.294597 4253 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:55.294734 4253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:00:55.295213 4253 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:00:55.295378 4253 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:00:55.295476 4253 fault_injection.cc:59] injecting fault for kudu (pid 4253): FLAGS_fault_before_start (process will exit)
I20260812 20:00:55.318518 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:46837
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285
--webserver_port=37475
--webserver_interface=127.25.14.1
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295 with env {}
W20260812 20:00:55.736364 4278 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:00:55.736742 4278 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:00:55.736946 4278 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:00:55.746073 4278 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:00:55.746315 4278 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:00:55.757206 4278 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40513,127.25.14.21:35649,127.25.14.22:33863,127.25.14.23:38295,127.25.14.24:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:46837
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=37475
--tserver_master_addrs=127.25.14.62:35047,127.25.14.61:46621,127.25.14.60:33285,127.25.14.59:38295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:00:55.759085 4278 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:00:55.761211 4278 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:00:55.772796 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:55.772764 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:00:55.773908 4291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:00:55.774273 4278 server_base.cc:1061] running on GCE node
I20260812 20:00:55.775002 4278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:00:55.776332 4278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:00:55.779735 4278 hybrid_clock.cc:648] HybridClock initialized: now 1786564855779530 us; error 139 us; skew 500 ppm
I20260812 20:00:55.780224 4278 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:00:55.783192 4278 webserver.cc:533] Webserver started at http://127.25.14.1:37475/ using document root <none> and password file <none>
I20260812 20:00:55.784114 4278 fs_manager.cc:362] Metadata directory not provided
I20260812 20:00:55.784252 4278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:00:55.790892 4278 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:00:55.794750 4297 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:55.796568 4278 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:00:55.796865 4278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f549ed45eea8402b852efe2c379d6587"
format_stamp: "Formatted at 2026-08-12 20:00:51 on dist-test-slave-p3t5"
I20260812 20:00:55.797861 4278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:00:55.820168 4278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:00:55.821197 4278 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:00:55.821525 4278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:00:55.822793 4278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:00:55.824975 4278 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:00:55.825069 4278 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:55.825237 4278 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:00:55.825310 4278 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:00:55.865875 4278 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46837
I20260812 20:00:55.865919 4428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46837 every 8 connection(s)
I20260812 20:00:55.867645 4278 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:00:55.873718 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4278
I20260812 20:00:55.874068 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4278
I20260812 20:00:55.895174 4429 heartbeater.cc:344] Connected to a master server at 127.25.14.59:38295
I20260812 20:00:55.895710 4429 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:55.897231 4429 heartbeater.cc:507] Master 127.25.14.59:38295 requested a full tablet report, sending...
I20260812 20:00:55.899715 3731 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:55.900624 4432 heartbeater.cc:344] Connected to a master server at 127.25.14.61:46621
I20260812 20:00:55.900789 4432 heartbeater.cc:461] Registering TS with master...
I20260812 20:00:55.901479 4432 heartbeater.cc:507] Master 127.25.14.61:46621 requested a full tablet report, sending...
I20260812 20:00:55.902846 3216 ts_manager.cc:194] Re-registered known tserver with Master: f549ed45eea8402b852efe2c379d6587 (127.25.14.1:46837)
I20260812 20:00:55.905395 4424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:55.997813 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4278
I20260812 20:00:56.028152 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3970
W20260812 20:00:56.029219 4441 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:00:55.892490 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:44549 (local address 127.25.14.60:33285)
0812 20:00:55.901863 (+ 9373us) server_negotiation.cc:207] Beginning negotiation
0812 20:00:55.901869 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 20:00:55.901890 (+ 21us) server_negotiation.cc:408] Connection header received
0812 20:00:55.901953 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:00:55.901959 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:00:55.902066 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:00:55.902874 (+ 808us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:00:55.903124 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:00:55.905060 (+ 1936us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:00:56.001114 (+ 96054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:00:56.001792 (+ 678us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:00:56.001973 (+ 181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:00:56.002026 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:00:56.002034 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0812 20:00:56.002059 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:00:56.002101 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0812 20:00:56.002120 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0812 20:00:56.002209 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0812 20:00:56.027920 (+ 25711us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:44549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":9246,"thread_start_us":9134,"threads_started":1}
I20260812 20:00:56.038071 4048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:56.161860 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3970
I20260812 20:00:56.193037 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3175
I20260812 20:00:56.194486 3250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:56.306455 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3175
I20260812 20:00:56.337011 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3261
I20260812 20:00:56.338656 3330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:56.453871 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3261
I20260812 20:00:56.484306 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 3566
I20260812 20:00:56.485926 3765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:00:56.602734 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 3566
Shutting down hive metastore.
2026-08-12T20:00:57Z chronyd exiting
2026-08-12T20:00:57Z chronyd exiting
2026-08-12T20:00:57Z chronyd exiting
2026-08-12T20:00:57Z chronyd exiting
2026-08-12T20:00:57Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/5 (25838 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/6
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:00:57 dist-test-slave-p3t5 krb5kdc[4457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:00:57 dist-test-slave-p3t5 krb5kdc[4457](info): set up 2 sockets
Aug 12 20:00:57 dist-test-slave-p3t5 krb5kdc[4457](info): commencing operation
krb5kdc: starting...
W20260812 20:00:59.364466 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.229s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:00:59 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:00:59.802577 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35265
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:00.204218 4471 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:00.204694 4471 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:00.204906 4471 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:00.213913 4471 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:00.214121 4471 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:00.214260 4471 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:00.214349 4471 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:00.226106 4471 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:00.228061 4471 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:00.229780 4471 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:00.239189 4476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:00.239180 4477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:00.240839 4480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:00.241196 4471 server_base.cc:1061] running on GCE node
I20260812 20:01:00.242278 4471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:00.242498 4471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:00.242604 4471 hybrid_clock.cc:648] HybridClock initialized: now 1786564860242582 us; error 0 us; skew 500 ppm
I20260812 20:01:00.243016 4471 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:00 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:00.247071 4471 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:00.249377 4471 webserver.cc:533] Webserver started at http://127.25.14.62:42709/ using document root <none> and password file <none>
I20260812 20:01:00.250240 4471 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:00.250351 4471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:00.250672 4471 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:00.253238 4471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "5bb2769269fc40779533de43e413df34"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.254123 4471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "5bb2769269fc40779533de43e413df34"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.260366 4471 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:01:00.264806 4489 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:00.266834 4471 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 20:01:00.267108 4471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5bb2769269fc40779533de43e413df34"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.267359 4471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:00.307507 4471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:00.312059 4471 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:00.312422 4471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:00.335969 4471 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35265
I20260812 20:01:00.335983 4544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35265 every 8 connection(s)
I20260812 20:01:00.337984 4471 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:00.343024 4545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:00.347980 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4471
I20260812 20:01:00.348214 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:01:00.350342 4545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:00 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:00.378023 4545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:00.379640 4492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
W20260812 20:01:00.381700 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:38689: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:00.431226 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } attempt: 1
W20260812 20:01:00.433961 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:38689: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:00.502303 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } attempt: 2
W20260812 20:01:00.504716 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:38689: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:00.530704 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:38689
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:00.596057 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } attempt: 3
W20260812 20:01:00.598506 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:38689: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
I20260812 20:01:00.741827 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } attempt: 4
W20260812 20:01:00.744513 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:38689: Network error: Client connection negotiation failed: client connection to 127.25.14.61:38689: connect: Connection refused (error 111)
W20260812 20:01:00.930598 4556 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:00.931070 4556 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:00.931236 4556 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:00.940029 4556 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:00.940222 4556 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:00.940361 4556 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:00.940451 4556 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:00.951993 4556 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:00.953909 4556 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:00.955399 4556 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:00.964366 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:00.964535 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:00.965592 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:00.965853 4556 server_base.cc:1061] running on GCE node
I20260812 20:01:00.966846 4556 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:00.967064 4556 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:00.967180 4556 hybrid_clock.cc:648] HybridClock initialized: now 1786564860967154 us; error 0 us; skew 500 ppm
I20260812 20:01:00.967592 4556 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:00 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:00.971522 4556 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:01:00.973943 4556 webserver.cc:533] Webserver started at http://127.25.14.61:40595/ using document root <none> and password file <none>
I20260812 20:01:00.975008 4556 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:00.975174 4556 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:00.975610 4556 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:00.978348 4556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "532bc06f515540aaa90423d570e19c2e"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.979189 4556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "532bc06f515540aaa90423d570e19c2e"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.985944 4556 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 20:01:00.990065 4575 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:00.991634 4556 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:01:00.991865 4556 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "532bc06f515540aaa90423d570e19c2e"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:00.992077 4556 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:01.015107 4556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:01.020246 4556 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:01.020661 4556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:01.043799 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } attempt: 5
I20260812 20:01:01.043910 4556 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:38689
I20260812 20:01:01.043900 4627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:38689 every 8 connection(s)
I20260812 20:01:01.046057 4556 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:01:01.049172 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4556
I20260812 20:01:01.049408 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:01:01.052165 4630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:01.060467 4630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:01.077592 4545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:01.080201 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:01.091468 4630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:01.106065 4630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:01.107304 4579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
W20260812 20:01:01.109193 4630 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
I20260812 20:01:01.131480 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 1
W20260812 20:01:01.133965 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:01.158649 4630 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 1
W20260812 20:01:01.161164 4630 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:01:01.202237 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 2
W20260812 20:01:01.204993 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:01.229457 4630 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 2
I20260812 20:01:01.230362 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35241
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:01.231885 4630 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
I20260812 20:01:01.311316 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 3
W20260812 20:01:01.313838 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
I20260812 20:01:01.323166 4630 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 3
W20260812 20:01:01.325559 4630 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
I20260812 20:01:01.468889 4630 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 4
W20260812 20:01:01.471460 4630 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
I20260812 20:01:01.491137 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 4
W20260812 20:01:01.493512 4545 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35241: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35241: connect: Connection refused (error 111)
W20260812 20:01:01.623757 4639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:01.624181 4639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:01.624292 4639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:01.634253 4639 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:01.634384 4639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:01.634454 4639 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:01.634509 4639 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:01.647394 4639 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:01.649371 4639 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:01.651136 4639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:01.661566 4646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:01.661495 4647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:01.662949 4649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:01.663643 4639 server_base.cc:1061] running on GCE node
I20260812 20:01:01.664608 4639 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:01.664826 4639 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:01.664916 4639 hybrid_clock.cc:648] HybridClock initialized: now 1786564861664896 us; error 0 us; skew 500 ppm
I20260812 20:01:01.665287 4639 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:01.669334 4639 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:01:01.671977 4639 webserver.cc:533] Webserver started at http://127.25.14.60:35707/ using document root <none> and password file <none>
I20260812 20:01:01.672768 4639 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:01.672874 4639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:01.673211 4639 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:01.675971 4639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "eb6741c12dab4cac8b24566d69b1372b"
format_stamp: "Formatted at 2026-08-12 20:01:01 on dist-test-slave-p3t5"
I20260812 20:01:01.676772 4639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "eb6741c12dab4cac8b24566d69b1372b"
format_stamp: "Formatted at 2026-08-12 20:01:01 on dist-test-slave-p3t5"
I20260812 20:01:01.682766 4639 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260812 20:01:01.686785 4655 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:01.688428 4639 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:01:01.688673 4639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "eb6741c12dab4cac8b24566d69b1372b"
format_stamp: "Formatted at 2026-08-12 20:01:01 on dist-test-slave-p3t5"
I20260812 20:01:01.688916 4639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:01.712702 4639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:01.717226 4639 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:01.717562 4639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:01.740542 4639 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:35241
I20260812 20:01:01.740567 4716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:35241 every 8 connection(s)
I20260812 20:01:01.742441 4639 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:01:01.743021 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4639
I20260812 20:01:01.743222 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:01:01.749517 4718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:01.761650 4718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:01.770735 4630 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 5
I20260812 20:01:01.770848 4545 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } attempt: 5
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:01.795708 4718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:01.815182 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Bootstrap starting.
I20260812 20:01:01.815184 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: Bootstrap starting.
I20260812 20:01:01.819466 4545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:01.819482 4630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:01.820835 4718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:01.821265 4545 log.cc:826] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:01.821518 4630 log.cc:826] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:01.828706 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: No bootstrap required, opened a new log
I20260812 20:01:01.828699 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: No bootstrap required, opened a new log
Aug 12 20:01:01 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:01.836171 4545 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.836582 4545 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:01.836615 4630 raft_consensus.cc:359] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.836783 4545 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb2769269fc40779533de43e413df34, State: Initialized, Role: FOLLOWER
I20260812 20:01:01.836941 4630 raft_consensus.cc:385] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:01.837088 4630 raft_consensus.cc:740] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532bc06f515540aaa90423d570e19c2e, State: Initialized, Role: FOLLOWER
I20260812 20:01:01.837597 4545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.837870 4630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.839008 4733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:01.841780 4733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:01.841012 4630 sys_catalog.cc:565] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:01.845110 4732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:01.845441 4732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:01.847463 4545 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:01.848343 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: Bootstrap starting.
I20260812 20:01:01.853299 4718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:01.855086 4718 log.cc:826] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:01.859120 4718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: No bootstrap required, opened a new log
W20260812 20:01:01.867219 4757 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:01.867815 4757 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:01.868340 4718 raft_consensus.cc:359] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.868840 4718 raft_consensus.cc:385] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:01.869014 4718 raft_consensus.cc:740] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6741c12dab4cac8b24566d69b1372b, State: Initialized, Role: FOLLOWER
I20260812 20:01:01.869933 4718 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.860945 4556 master_runner.cc:187] Error getting master registration for 127.25.14.62:35265: OK, instance_id { permanent_uuid: "5bb2769269fc40779533de43e413df34" instance_seqno: 1786564860329674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:01.872864 4556 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:01.874023 4766 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:01.874656 4766 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:01.875355 4718 sys_catalog.cc:565] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:01.893165 4763 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:01.893708 4763 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:01.888180 4471 master_runner.cc:187] Error getting master registration for 127.25.14.60:35241: OK, instance_id { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" instance_seqno: 1786564861734199 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:01.900003 4471 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:01.914642 4732 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:01:01.914960 4732 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.917068 4732 leader_election.cc:290] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 532bc06f515540aaa90423d570e19c2e (127.25.14.61:38689), eb6741c12dab4cac8b24566d69b1372b (127.25.14.60:35241)
I20260812 20:01:01.918104 4689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb2769269fc40779533de43e413df34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6741c12dab4cac8b24566d69b1372b" is_pre_election: true
I20260812 20:01:01.918771 4689 raft_consensus.cc:2468] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb2769269fc40779533de43e413df34 in term 0.
I20260812 20:01:01.919593 4493 leader_election.cc:304] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bb2769269fc40779533de43e413df34, eb6741c12dab4cac8b24566d69b1372b; no voters:
I20260812 20:01:01.919927 4603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb2769269fc40779533de43e413df34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532bc06f515540aaa90423d570e19c2e" is_pre_election: true
I20260812 20:01:01.920073 4732 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:01:01.920230 4732 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:01:01.920397 4732 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:01.920487 4603 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb2769269fc40779533de43e413df34 in term 0.
I20260812 20:01:01.923940 4732 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.930344 4639 master_runner.cc:233] Existing masters have differing terms: 1 vs 0
I20260812 20:01:01.930645 4639 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:01.931828 4603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb2769269fc40779533de43e413df34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532bc06f515540aaa90423d570e19c2e"
I20260812 20:01:01.932070 4603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:01.936184 4603 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb2769269fc40779533de43e413df34 in term 1.
I20260812 20:01:01.937394 4732 leader_election.cc:290] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [CANDIDATE]: Term 1 election: Requested vote from peers 532bc06f515540aaa90423d570e19c2e (127.25.14.61:38689), eb6741c12dab4cac8b24566d69b1372b (127.25.14.60:35241)
I20260812 20:01:01.937582 4492 leader_election.cc:304] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 532bc06f515540aaa90423d570e19c2e, 5bb2769269fc40779533de43e413df34; no voters:
I20260812 20:01:01.938277 4689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb2769269fc40779533de43e413df34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6741c12dab4cac8b24566d69b1372b"
I20260812 20:01:01.938524 4689 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:01.942384 4689 raft_consensus.cc:2468] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb2769269fc40779533de43e413df34 in term 1.
W20260812 20:01:01.942668 4780 catalog_manager.cc:1594] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:01.942813 4780 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:01.943554 4732 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:01:01.954864 4732 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb2769269fc40779533de43e413df34, State: Running, Role: LEADER
I20260812 20:01:01.955669 4732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } }
I20260812 20:01:01.966990 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb2769269fc40779533de43e413df34. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:01.967437 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
W20260812 20:01:01.970093 4661 tablet.cc:2406] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:01.976519 4784 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:01:01.985404 4784 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:01:02.007601 4689 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Refusing update from remote peer 5bb2769269fc40779533de43e413df34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:02.008996 4782 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:02.016708 4603 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Refusing update from remote peer 5bb2769269fc40779533de43e413df34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:02.017982 4782 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 20:01:02.023936 4581 tablet.cc:2406] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:01:02.068935 4495 tablet.cc:2406] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:02.071157 4733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb2769269fc40779533de43e413df34. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.071163 4766 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb2769269fc40779533de43e413df34. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.071465 4733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:02.071478 4766 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:02.079895 4787 mvcc.cc:204] Tried to move back new op lower bound from 7317769674748559360 to 7317769674577387520. Current Snapshot: MvccSnapshot[applied={T|T < 7317769674748559360}]
I20260812 20:01:02.081841 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.082136 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:02.082413 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.082579 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:02.086591 4786 mvcc.cc:204] Tried to move back new op lower bound from 7317769674748559360 to 7317769674577387520. Current Snapshot: MvccSnapshot[applied={T|T < 7317769674748559360}]
I20260812 20:01:02.091130 4766 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.091418 4766 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:02.094970 4784 catalog_manager.cc:1383] Generated new cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:02.095112 4784 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:01:02.111783 4733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } }
I20260812 20:01:02.112041 4733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:02.113329 4789 mvcc.cc:204] Tried to move back new op lower bound from 7317769674748559360 to 7317769674577387520. Current Snapshot: MvccSnapshot[applied={T|T < 7317769674748559360 or (T in {7317769674748559360})}]
I20260812 20:01:02.123515 4784 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:01:02.125509 4784 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:01:02.153877 4784 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Generated new TSK 0
I20260812 20:01:02.159965 4784 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:02.293385 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.106s user 0.000s sys 0.036s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:02.433003 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.139s user 0.000s sys 0.037s
I20260812 20:01:02.434334 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:02.870062 4757 catalog_manager.cc:1295] Loaded cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:02.870168 4757 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: loading cluster ID for follower catalog manager: success
I20260812 20:01:02.872836 4757 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: acquiring CA information for follower catalog manager: success
W20260812 20:01:02.872816 4801 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:02.873296 4801 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:02.873461 4801 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 20:01:02.875128 4757 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 20:01:02.883289 4801 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:02.883543 4801 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:02.895046 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:02.896981 4801 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:02.898633 4801 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:02.909294 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:02.909317 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:02.909945 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:02.910692 4801 server_base.cc:1061] running on GCE node
I20260812 20:01:02.911424 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:02.911636 4801 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:02.911764 4801 hybrid_clock.cc:648] HybridClock initialized: now 1786564862911733 us; error 0 us; skew 500 ppm
I20260812 20:01:02.912127 4801 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:02 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:02.915925 4801 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:02.917919 4801 webserver.cc:533] Webserver started at http://127.25.14.1:36181/ using document root <none> and password file <none>
I20260812 20:01:02.918672 4801 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:02.918773 4801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:02.919101 4801 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:02.921506 4801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:02.922327 4801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:02.928048 4801 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 20:01:02.931639 4816 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:02.933135 4801 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:01:02.933333 4801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:02.933553 4801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:02.945196 4780 catalog_manager.cc:1295] Loaded cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:02.945322 4780 catalog_manager.cc:1588] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: loading cluster ID for follower catalog manager: success
I20260812 20:01:02.949098 4780 catalog_manager.cc:1610] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: acquiring CA information for follower catalog manager: success
I20260812 20:01:02.951700 4780 catalog_manager.cc:1638] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:02.957334 4801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:02.967197 4801 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:02.967540 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:02.968637 4801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:02.970510 4801 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:02.970633 4801 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:02.970791 4801 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:02.970876 4801 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:03.008569 4801 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40313
I20260812 20:01:03.008581 4948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40313 every 8 connection(s)
I20260812 20:01:03.010217 4801 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:03.019101 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4801
I20260812 20:01:03.019407 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:03.064472 4949 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35241
I20260812 20:01:03.064868 4950 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35265
I20260812 20:01:03.065048 4950 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:03.064899 4949 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:03.066393 4949 heartbeater.cc:507] Master 127.25.14.60:35241 requested a full tablet report, sending...
I20260812 20:01:03.066368 4950 heartbeater.cc:507] Master 127.25.14.62:35265 requested a full tablet report, sending...
I20260812 20:01:03.068039 4951 heartbeater.cc:344] Connected to a master server at 127.25.14.61:38689
I20260812 20:01:03.068200 4951 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:03.068758 4951 heartbeater.cc:507] Master 127.25.14.61:38689 requested a full tablet report, sending...
I20260812 20:01:03.069247 4506 ts_manager.cc:194] Registered new tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:03.069281 4677 ts_manager.cc:194] Registered new tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:03.070658 4593 ts_manager.cc:194] Registered new tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:03.070967 4506 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:59091
I20260812 20:01:03.083626 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:03.256973 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:39139
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:03.647535 4967 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:03.647971 4967 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:03.648131 4967 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:03.657253 4967 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:03.657430 4967 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:03.657555 4967 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:03.657663 4967 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:03.669273 4967 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:03.671133 4967 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:03.672650 4967 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:03.682386 4973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:03.682459 4976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:03.682407 4974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:03.683387 4967 server_base.cc:1061] running on GCE node
I20260812 20:01:03.684296 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:03.684514 4967 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:03.684628 4967 hybrid_clock.cc:648] HybridClock initialized: now 1786564863684602 us; error 0 us; skew 500 ppm
I20260812 20:01:03.685019 4967 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:03.689108 4967 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:03.691385 4967 webserver.cc:533] Webserver started at http://127.25.14.59:39865/ using document root <none> and password file <none>
I20260812 20:01:03.692173 4967 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:03.692274 4967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:03.692607 4967 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:03.695257 4967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "9d214abd570b40b1906137de00d8e270"
format_stamp: "Formatted at 2026-08-12 20:01:03 on dist-test-slave-p3t5"
I20260812 20:01:03.696079 4967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "9d214abd570b40b1906137de00d8e270"
format_stamp: "Formatted at 2026-08-12 20:01:03 on dist-test-slave-p3t5"
I20260812 20:01:03.702296 4967 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:01:03.706286 4982 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:03.707872 4967 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:01:03.708097 4967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "9d214abd570b40b1906137de00d8e270"
format_stamp: "Formatted at 2026-08-12 20:01:03 on dist-test-slave-p3t5"
I20260812 20:01:03.708287 4967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:03.731244 4967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:03.736084 4967 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:03.736438 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:03.762987 5042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:39139 every 8 connection(s)
I20260812 20:01:03.763001 4967 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:39139
I20260812 20:01:03.764962 4967 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:03.769604 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 4967
I20260812 20:01:03.770004 5043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:03.779563 5043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:03.802260 5043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:03.817315 5043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:03.832342 5043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:03.850690 5043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Bootstrap starting.
I20260812 20:01:03.854658 5043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:03.855878 5043 log.cc:826] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:03.858839 5043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: No bootstrap required, opened a new log
I20260812 20:01:03.864343 5043 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } }
I20260812 20:01:03.864756 5043 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:03.864903 5043 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d214abd570b40b1906137de00d8e270, State: Initialized, Role: FOLLOWER
I20260812 20:01:03.865594 5043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } }
I20260812 20:01:03.866783 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } } }
I20260812 20:01:03.867172 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:03.867923 5043 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:03.879204 5064 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:03.879455 5064 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:03.887602 4967 master_runner.cc:429] Detected that this master 9d214abd570b40b1906137de00d8e270 is joining an existing cluster
I20260812 20:01:03.887784 4967 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:01:03.888903 4967 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:01:03.889325 4967 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:01:03.889549 4967 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: stopping tablet replica
I20260812 20:01:03.891340 4967 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:39139
Aug 12 20:01:03 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:03.954913 4506 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:39139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d214abd570b40b1906137de00d8e270"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 39139
}
attrs {
promote: true
}
}
dest_uuid: "5bb2769269fc40779533de43e413df34"
cas_config_opid_index: -1
I20260812 20:01:03.956362 4506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:03.960019 4602 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Refusing update from remote peer 5bb2769269fc40779533de43e413df34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:01:03.960983 4490 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 -> Peer 9d214abd570b40b1906137de00d8e270 (127.25.14.59:39139): Couldn't send request to peer 9d214abd570b40b1906137de00d8e270. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:03.961045 4964 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:03.961217 4688 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Refusing update from remote peer 5bb2769269fc40779533de43e413df34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:01:03.961810 4964 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:03.965421 4964 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d214abd570b40b1906137de00d8e270 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:03.965992 4602 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d214abd570b40b1906137de00d8e270 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:03.972169 4964 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:39139
I20260812 20:01:03.973385 4967 master.cc:562] Master@127.25.14.59:39139 shutting down...
I20260812 20:01:03.995330 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:03.995788 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:04.011577 4967 master.cc:584] Master@127.25.14.59:39139 shutdown complete.
I20260812 20:01:04.011961 4967 master_runner.cc:306] Clearing existing system tablet
I20260812 20:01:04.012490 4967 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:04.012656 4967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:04.014557 4688 raft_consensus.cc:2955] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d214abd570b40b1906137de00d8e270 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.015625 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.015966 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:04.018136 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.018447 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:04.018774 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.018977 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:04.019908 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.020321 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:04.021291 4967 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.002s sys 0.001s
I20260812 20:01:04.023119 5090 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:04.023620 4967 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:04.023909 4967 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 20:01:04.024041 4967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "9d214abd570b40b1906137de00d8e270"
format_stamp: "Formatted at 2026-08-12 20:01:03 on dist-test-slave-p3t5"
I20260812 20:01:04.027681 4967 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:01:04.032616 4967 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:01:04.032876 4967 log.cc:1199] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:01:04.033345 4967 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Deleting consensus metadata
I20260812 20:01:04.033700 4967 master_runner.cc:316] Copying system tablet from 127.25.14.62:35265
I20260812 20:01:04.036325 4967 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.62:35265
I20260812 20:01:04.050479 4528 tablet_copy_service.cc:140] P 5bb2769269fc40779533de43e413df34: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d214abd570b40b1906137de00d8e270 ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54990)
I20260812 20:01:04.050820 4528 tablet_copy_service.cc:161] P 5bb2769269fc40779533de43e413df34: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d214abd570b40b1906137de00d8e270 at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54990: session id = 9d214abd570b40b1906137de00d8e270-00000000000000000000000000000000
I20260812 20:01:04.054116 4528 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:01:04.056680 4967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:01:04.062474 4967 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: tablet copy: Starting download of 0 data blocks...
I20260812 20:01:04.062844 4967 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: tablet copy: Starting download of 1 WAL segments...
I20260812 20:01:04.066246 4967 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:01:04.070272 4528 tablet_copy_service.cc:342] P 5bb2769269fc40779533de43e413df34: Request end of tablet copy session 9d214abd570b40b1906137de00d8e270-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54990
I20260812 20:01:04.070559 4528 tablet_copy_service.cc:434] P 5bb2769269fc40779533de43e413df34: ending tablet copy session 9d214abd570b40b1906137de00d8e270-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d214abd570b40b1906137de00d8e270
I20260812 20:01:04.073952 4950 heartbeater.cc:499] Master 127.25.14.62:35265 was elected leader, sending a full tablet report...
I20260812 20:01:04.084621 4967 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:04.087831 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:04.088673 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:04.090022 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:04.095841 4967 server_base.cc:1061] running on GCE node
I20260812 20:01:04.096271 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:04.096364 4967 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:04.096443 4967 hybrid_clock.cc:648] HybridClock initialized: now 1786564864096397 us; error 0 us; skew 500 ppm
I20260812 20:01:04.096660 4967 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:04.099251 4967 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:04.100598 4967 webserver.cc:533] Webserver started at http://127.25.14.59:46813/ using document root <none> and password file <none>
I20260812 20:01:04.101037 4967 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:04.101145 4967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:04.104513 4967 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 20:01:04.106650 5108 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:04.107209 4967 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 20:01:04.107360 4967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "9d214abd570b40b1906137de00d8e270"
format_stamp: "Formatted at 2026-08-12 20:01:03 on dist-test-slave-p3t5"
I20260812 20:01:04.107515 4967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:04.120609 4967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:04.124284 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:04.147011 4967 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:39139
I20260812 20:01:04.147053 5170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:39139 every 8 connection(s)
I20260812 20:01:04.148034 4967 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:04.151913 5171 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:04.154595 5171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Bootstrap starting.
I20260812 20:01:04.172411 5171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:04.173045 5171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: Bootstrap complete.
I20260812 20:01:04.174346 5171 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:04.174602 5171 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d214abd570b40b1906137de00d8e270, State: Initialized, Role: LEARNER
I20260812 20:01:04.174892 5171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:04.175809 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.176110 5171 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:04.176245 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:04.189843 5193 catalog_manager.cc:1295] Loaded cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:04.189981 5193 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: loading cluster ID for follower catalog manager: success
I20260812 20:01:04.192780 5193 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: acquiring CA information for follower catalog manager: success
I20260812 20:01:04.195194 5193 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:04.230810 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4801
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:01:04.250883 4944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:04.341341 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4801
I20260812 20:01:04.367004 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40313
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--webserver_port=36181
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:04.571714 5142 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:01:04.572733 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb2769269fc40779533de43e413df34. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:04.573071 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:01:04.773836 5196 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:04.774259 5196 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:04.774439 5196 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:04.783640 5196 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:04.783936 5196 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:04.795663 5196 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36181
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:04.797753 5196 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:04.799360 5196 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:04.811211 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:04.811192 5203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:04.811446 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:04.812711 5196 server_base.cc:1061] running on GCE node
I20260812 20:01:04.813771 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:04.814055 5196 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:04.814210 5196 hybrid_clock.cc:648] HybridClock initialized: now 1786564864814150 us; error 0 us; skew 500 ppm
I20260812 20:01:04.814612 5196 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:04.818729 5196 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:04.820690 5196 webserver.cc:533] Webserver started at http://127.25.14.1:36181/ using document root <none> and password file <none>
I20260812 20:01:04.821529 5196 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:04.821756 5196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:04.828661 5196 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260812 20:01:04.832321 5212 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:04.834096 5196 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:04.834332 5196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:04.835212 5196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:04.855886 5196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:04.861076 5196 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:04.861438 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:04.862694 5196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:04.864642 5196 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:04.864745 5196 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:04.864935 5196 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:04.865046 5196 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:04.906772 5196 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40313
I20260812 20:01:04.906893 5348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40313 every 8 connection(s)
I20260812 20:01:04.908620 5196 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:04.911283 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5196
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:04 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564864, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:04.954340 5349 heartbeater.cc:344] Connected to a master server at 127.25.14.59:39139
I20260812 20:01:04.954878 5349 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:04.955143 5352 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35265
I20260812 20:01:04.955312 5352 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:04.955782 5353 heartbeater.cc:344] Connected to a master server at 127.25.14.61:38689
I20260812 20:01:04.955986 5353 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:04.956317 5352 heartbeater.cc:507] Master 127.25.14.62:35265 requested a full tablet report, sending...
I20260812 20:01:04.956351 5349 heartbeater.cc:507] Master 127.25.14.59:39139 requested a full tablet report, sending...
I20260812 20:01:04.956511 5353 heartbeater.cc:507] Master 127.25.14.61:38689 requested a full tablet report, sending...
I20260812 20:01:04.958635 4593 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:04.958962 5130 ts_manager.cc:194] Registered new tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:04.959479 5351 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35241
I20260812 20:01:04.959638 5351 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:04.960215 5351 heartbeater.cc:507] Master 127.25.14.60:35241 requested a full tablet report, sending...
I20260812 20:01:04.960753 4506 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:04.961562 4677 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:04.961875 4506 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:43755
I20260812 20:01:04.967906 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:04.978760 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:04.978950 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4471
I20260812 20:01:04.979992 4538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:05.110615 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bb2769269fc40779533de43e413df34" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:05.110944 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:05.111173 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4471
I20260812 20:01:05.146478 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35265
--webserver_interface=127.25.14.62
--webserver_port=42709
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:05.548959 5368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:05.549371 5368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:05.549551 5368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:05.558653 5368 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:05.558776 5368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:05.558885 5368 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:05.558959 5368 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:05.570356 5368 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=42709
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:05.572394 5368 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:05.573972 5368 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:05.586186 5375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:05.586488 5378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:05.586134 5376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:05.587234 5368 server_base.cc:1061] running on GCE node
I20260812 20:01:05.588122 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:05.588346 5368 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:05.588461 5368 hybrid_clock.cc:648] HybridClock initialized: now 1786564865588434 us; error 0 us; skew 500 ppm
I20260812 20:01:05.588855 5368 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:05 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564865, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:05.592723 5368 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:05.595196 5368 webserver.cc:533] Webserver started at http://127.25.14.62:42709/ using document root <none> and password file <none>
I20260812 20:01:05.596022 5368 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:05.596189 5368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:05.602366 5368 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:01:05.605738 5384 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:05.607383 5368 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:01:05.607604 5368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5bb2769269fc40779533de43e413df34"
format_stamp: "Formatted at 2026-08-12 20:01:00 on dist-test-slave-p3t5"
I20260812 20:01:05.608420 5368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:05.629235 5368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:05.633672 5368 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:05.634011 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:05.657879 5368 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35265
I20260812 20:01:05.657903 5442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35265 every 8 connection(s)
I20260812 20:01:05.659746 5368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:05.660801 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5368
I20260812 20:01:05.661185 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5196
I20260812 20:01:05.669432 5443 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:05.673555 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Bootstrap starting.
I20260812 20:01:05.679975 5342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:05.694841 5443 log.cc:826] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:05.705359 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:05.706053 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: Bootstrap complete.
I20260812 20:01:05.713788 5443 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:05.714676 5443 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb2769269fc40779533de43e413df34, State: Initialized, Role: FOLLOWER
I20260812 20:01:05.715528 5443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:05.716491 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:05.716840 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:05.717868 5443 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:05.730422 5462 catalog_manager.cc:1295] Loaded cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:05.730574 5462 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: loading cluster ID for follower catalog manager: success
I20260812 20:01:05.735886 5462 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: acquiring CA information for follower catalog manager: success
I20260812 20:01:05.741194 5462 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 20:01:05 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564865, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:05.796363 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5196
I20260812 20:01:05.808136 5368 master_runner.cc:264] No leader master found from master 5bb2769269fc40779533de43e413df34
I20260812 20:01:05.808337 5368 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:05.824281 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40313
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--webserver_port=36181
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:06.263002 5470 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:06.263360 5470 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:06.263455 5470 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:06.272462 5470 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:06.272671 5470 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:06.284822 5470 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36181
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:06.286826 5470 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:06.288312 5470 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:06.300041 5478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:06.300386 5476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:06.299994 5475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:06.301126 5470 server_base.cc:1061] running on GCE node
I20260812 20:01:06.301975 5470 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:06.302170 5470 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:06.302284 5470 hybrid_clock.cc:648] HybridClock initialized: now 1786564866302240 us; error 0 us; skew 500 ppm
I20260812 20:01:06.302711 5470 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:06 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:06.306775 5470 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:06.308956 5470 webserver.cc:533] Webserver started at http://127.25.14.1:36181/ using document root <none> and password file <none>
I20260812 20:01:06.309914 5470 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:06.310053 5470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:06.316498 5470 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 20:01:06.319789 5487 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:06.321452 5470 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:01:06.321699 5470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:06.322597 5470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:06.358991 5470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:06.363281 5470 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:06.363600 5470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:06.364709 5470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:06.366616 5470 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:06.366746 5470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:06.366894 5470 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:06.366997 5470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:06.405666 5470 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40313
I20260812 20:01:06.405781 5613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40313 every 8 connection(s)
I20260812 20:01:06.407425 5470 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:06.408215 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5470
Aug 12 20:01:06 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:06 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:06 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:06.454521 5630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5bb2769269fc40779533de43e413df34)
I20260812 20:01:06.454830 5630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:06.461103 5614 heartbeater.cc:344] Connected to a master server at 127.25.14.59:39139
I20260812 20:01:06.461643 5614 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:06.462960 5614 heartbeater.cc:507] Master 127.25.14.59:39139 requested a full tablet report, sending...
Aug 12 20:01:06 dist-test-slave-p3t5 krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:06.466094 5130 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:06.467059 5618 heartbeater.cc:344] Connected to a master server at 127.25.14.61:38689
I20260812 20:01:06.467231 5618 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:06.467841 5618 heartbeater.cc:507] Master 127.25.14.61:38689 requested a full tablet report, sending...
I20260812 20:01:06.469419 4593 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:06.469864 4688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "532bc06f515540aaa90423d570e19c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "eb6741c12dab4cac8b24566d69b1372b" is_pre_election: true
I20260812 20:01:06.471714 5630 leader_election.cc:290] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5bb2769269fc40779533de43e413df34 (127.25.14.62:35265), eb6741c12dab4cac8b24566d69b1372b (127.25.14.60:35241)
I20260812 20:01:06.472477 5616 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35241
I20260812 20:01:06.472672 5616 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:06.473356 5616 heartbeater.cc:507] Master 127.25.14.60:35241 requested a full tablet report, sending...
I20260812 20:01:06.474850 4677 ts_manager.cc:194] Re-registered known tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:06.477142 5617 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35265
I20260812 20:01:06.477306 5617 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:06.477957 5617 heartbeater.cc:507] Master 127.25.14.62:35265 requested a full tablet report, sending...
I20260812 20:01:06.480278 5402 ts_manager.cc:194] Registered new tserver with Master: 0a6e03957d804a758e300149f29a0858 (127.25.14.1:40313)
I20260812 20:01:06.484283 5416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "532bc06f515540aaa90423d570e19c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5bb2769269fc40779533de43e413df34" is_pre_election: true
I20260812 20:01:06.484867 5416 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 532bc06f515540aaa90423d570e19c2e in term 1.
I20260812 20:01:06.485540 4576 leader_election.cc:304] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 532bc06f515540aaa90423d570e19c2e, 5bb2769269fc40779533de43e413df34; no voters: eb6741c12dab4cac8b24566d69b1372b
I20260812 20:01:06.486079 5630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 20:01:06.486230 5630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5bb2769269fc40779533de43e413df34)
I20260812 20:01:06.486338 5630 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 1 FOLLOWER]: Advancing to term 2
I20260812 20:01:06.488636 5630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:06.489478 5630 leader_election.cc:290] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [CANDIDATE]: Term 2 election: Requested vote from peers 5bb2769269fc40779533de43e413df34 (127.25.14.62:35265), eb6741c12dab4cac8b24566d69b1372b (127.25.14.60:35241)
I20260812 20:01:06.490099 5416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "532bc06f515540aaa90423d570e19c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5bb2769269fc40779533de43e413df34"
I20260812 20:01:06.490197 4688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "532bc06f515540aaa90423d570e19c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "eb6741c12dab4cac8b24566d69b1372b"
I20260812 20:01:06.490356 5416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 1 FOLLOWER]: Advancing to term 2
I20260812 20:01:06.494840 5416 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 532bc06f515540aaa90423d570e19c2e in term 2.
I20260812 20:01:06.495483 4576 leader_election.cc:304] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 532bc06f515540aaa90423d570e19c2e, 5bb2769269fc40779533de43e413df34; no voters: eb6741c12dab4cac8b24566d69b1372b
I20260812 20:01:06.495923 5630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 20:01:06.496445 5630 raft_consensus.cc:697] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 2 LEADER]: Becoming Leader. State: Replica: 532bc06f515540aaa90423d570e19c2e, State: Running, Role: LEADER
I20260812 20:01:06.496974 5630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:06.499603 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532bc06f515540aaa90423d570e19c2e. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.499926 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.500773 5636 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:01:06.503501 5636 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:01:06.504700 5636 catalog_manager.cc:1295] Loaded cluster ID: 304e7383ebba4beda246601956345631
I20260812 20:01:06.504809 5636 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:01:06.506524 5636 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:01:06.507758 5636 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: Loaded TSK: 0
I20260812 20:01:06.509114 5636 catalog_manager.cc:1550] Initializing in-progress tserver states...
W20260812 20:01:06.529148 25656 mini_kdc.cc:336] Time spent kdestroy: real 0.121s user 0.006s sys 0.034s
I20260812 20:01:06.533085 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5470
W20260812 20:01:06.534617 5639 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:06.530627 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.25.14.1:40313)
0812 20:01:06.531331 (+ 704us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:06.531339 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:06.531378 (+ 39us) server_negotiation.cc:408] Connection header received
0812 20:01:06.531448 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:06.531456 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:06.531683 (+ 227us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:06.531869 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:06.532895 (+ 1026us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49020: BlockingRecv error: recv got EOF from 127.0.0.1:49020 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":368,"threads_started":1}
W20260812 20:01:06.536024 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5470: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40313: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:06.536204 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5470
I20260812 20:01:06.565069 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40313
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--webserver_port=36181
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:06.610872 5642 raft_consensus.cc:493] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5bb2769269fc40779533de43e413df34)
I20260812 20:01:06.611195 5642 raft_consensus.cc:515] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } }
I20260812 20:01:06.612970 5642 leader_election.cc:290] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5bb2769269fc40779533de43e413df34 (127.25.14.62:35265), 532bc06f515540aaa90423d570e19c2e (127.25.14.61:38689)
I20260812 20:01:06.613915 4602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb6741c12dab4cac8b24566d69b1372b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "532bc06f515540aaa90423d570e19c2e" is_pre_election: true
I20260812 20:01:06.620455 5416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb6741c12dab4cac8b24566d69b1372b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5bb2769269fc40779533de43e413df34" is_pre_election: true
I20260812 20:01:06.620857 5416 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6741c12dab4cac8b24566d69b1372b in current term 2: Already voted for candidate 532bc06f515540aaa90423d570e19c2e in this term.
I20260812 20:01:06.621730 4656 leader_election.cc:304] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6741c12dab4cac8b24566d69b1372b; no voters: 532bc06f515540aaa90423d570e19c2e, 5bb2769269fc40779533de43e413df34
I20260812 20:01:06.622166 5642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 1 FOLLOWER]: Advancing to term 2
I20260812 20:01:06.624445 5642 raft_consensus.cc:2749] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 20:01:06.884470 5416 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 2 FOLLOWER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260812 20:01:06.885504 5634 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:06.890223 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532bc06f515540aaa90423d570e19c2e. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.890614 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.893349 4688 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 2 FOLLOWER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260812 20:01:06.893326 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.893659 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.894359 5647 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:06.902665 5142 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 1 LEARNER]: Advancing to term 2
I20260812 20:01:06.903245 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.903545 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.905695 5142 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 2 LEARNER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260812 20:01:06.907048 5630 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:06.914059 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532bc06f515540aaa90423d570e19c2e. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.914057 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.914788 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.914670 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.916499 5657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.916985 5652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532bc06f515540aaa90423d570e19c2e. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: true } } }
I20260812 20:01:06.917347 5652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:06.917377 5630 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e: attempting to promote NON_VOTER 9d214abd570b40b1906137de00d8e270 to VOTER
I20260812 20:01:06.916989 5657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:06.918587 5630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } }
I20260812 20:01:06.920912 4688 raft_consensus.cc:1275] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 2 FOLLOWER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260812 20:01:06.921003 5142 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 2 LEARNER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260812 20:01:06.921231 5416 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 2 FOLLOWER]: Refusing update from remote peer 532bc06f515540aaa90423d570e19c2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260812 20:01:06.921846 5634 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 20:01:06.922230 5647 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 20:01:06.922906 5634 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 20:01:06.925808 5634 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 9d214abd570b40b1906137de00d8e270 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.927280 5416 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 9d214abd570b40b1906137de00d8e270 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.928926 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.929066 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.929598 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.929807 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.930063 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.930254 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.930192 4688 raft_consensus.cc:2955] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 9d214abd570b40b1906137de00d8e270 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.931108 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.931324 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532bc06f515540aaa90423d570e19c2e [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:06.932036 5142 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 9d214abd570b40b1906137de00d8e270 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.933966 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.934274 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2769269fc40779533de43e413df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.935783 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.936018 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6741c12dab4cac8b24566d69b1372b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:06.936996 5657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "532bc06f515540aaa90423d570e19c2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5bb2769269fc40779533de43e413df34" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35265 } } peers { permanent_uuid: "532bc06f515540aaa90423d570e19c2e" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 38689 } } peers { permanent_uuid: "eb6741c12dab4cac8b24566d69b1372b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35241 } } peers { permanent_uuid: "9d214abd570b40b1906137de00d8e270" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39139 } attrs { promote: false } } }
I20260812 20:01:06.937268 5657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d214abd570b40b1906137de00d8e270 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:01:06.985811 5641 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:06.986274 5641 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:06.986465 5641 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:01:06.986579 5641 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:06.996335 5641 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:06.996636 5641 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:07.008108 5641 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36181
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:07.010120 5641 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:07.011615 5641 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:07.022841 5665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:07.022954 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:07.022794 5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:07.024519 5641 server_base.cc:1061] running on GCE node
I20260812 20:01:07.025154 5641 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:07.025300 5641 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:07.025393 5641 hybrid_clock.cc:648] HybridClock initialized: now 1786564867025351 us; error 0 us; skew 500 ppm
I20260812 20:01:07.025758 5641 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:07 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:07.029381 5641 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:07.031291 5641 webserver.cc:533] Webserver started at http://127.25.14.1:36181/ using document root <none> and password file <none>
I20260812 20:01:07.032141 5641 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:07.032239 5641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:07.038959 5641 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:01:07.042297 5674 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:07.044020 5641 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:01:07.044225 5641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:07.045037 5641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:07.066807 5641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:07.071488 5641 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:07.071831 5641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:07.072922 5641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:07.074816 5641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:07.074937 5641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:07.075098 5641 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:07.075176 5641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:01:07.075615 5641 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:01:07.075702 5641 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:01:07.075790 5641 fault_injection.cc:59] injecting fault for kudu (pid 5641): FLAGS_fault_before_start (process will exit)
I20260812 20:01:07.099422 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40313
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241
--webserver_port=36181
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:07.492530 5684 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:07.492897 5684 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:07.493093 5684 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:07.502329 5684 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:07.502604 5684 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:07.514196 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36181
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35265,127.25.14.61:38689,127.25.14.60:35241,127.25.14.59:39139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:07.516078 5684 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:07.517607 5684 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:07.529282 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:07.529305 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:07.529175 5693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:07.530340 5684 server_base.cc:1061] running on GCE node
I20260812 20:01:07.531100 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:07.531315 5684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:07.531430 5684 hybrid_clock.cc:648] HybridClock initialized: now 1786564867531402 us; error 0 us; skew 500 ppm
I20260812 20:01:07.531806 5684 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:07 dist-test-slave-p3t5 krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:07.535907 5684 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:07.537919 5684 webserver.cc:533] Webserver started at http://127.25.14.1:36181/ using document root <none> and password file <none>
I20260812 20:01:07.538770 5684 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:07.538939 5684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:07.545166 5684 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 20:01:07.548557 5701 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:07.550344 5684 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 20:01:07.550521 5684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "0a6e03957d804a758e300149f29a0858"
format_stamp: "Formatted at 2026-08-12 20:01:02 on dist-test-slave-p3t5"
I20260812 20:01:07.551275 5684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:07.587507 5684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:07.592069 5684 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:07.592329 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:07.593392 5684 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:07.595340 5684 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:07.595445 5684 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:07.595525 5684 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:07.595579 5684 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:07.633490 5684 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40313
I20260812 20:01:07.633608 5831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40313 every 8 connection(s)
I20260812 20:01:07.635322 5684 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:07.642366 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5684
I20260812 20:01:07.642800 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5684
W20260812 20:01:07.682090 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5684: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40313: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:07.682322 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5684
I20260812 20:01:07.708791 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5368
W20260812 20:01:07.709424 5838 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:07.639855 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58447 (local address 127.25.14.60:35241)
0812 20:01:07.674564 (+ 34709us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:07.674573 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:07.674597 (+ 24us) server_negotiation.cc:408] Connection header received
0812 20:01:07.674691 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:07.674700 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:07.674845 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:07.675042 (+ 197us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:07.679752 (+ 4710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:07.681671 (+ 1919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:07.708412 (+ 26741us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58447: BlockingRecv error: recv got EOF from 127.25.14.1:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":34557,"thread_start_us":34581,"threads_started":1}
W20260812 20:01:07.709470 5845 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:07.671531 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:33729 (local address 127.25.14.59:39139)
0812 20:01:07.686517 (+ 14986us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:07.686526 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:07.708380 (+ 21854us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:33729: BlockingRecv error: recv got EOF from 127.25.14.1:33729 (error 108)
Metrics: {"server-negotiator.queue_time_us":14805,"thread_start_us":15768,"threads_started":1}
W20260812 20:01:07.709685 5839 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:07.640245 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:56499 (local address 127.25.14.61:38689)
0812 20:01:07.674081 (+ 33836us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:07.674091 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:07.674126 (+ 35us) server_negotiation.cc:408] Connection header received
0812 20:01:07.708863 (+ 34737us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:56499: BlockingRecv error: recv got EOF from 127.25.14.1:56499 (error 108)
Metrics: {"server-negotiator.queue_time_us":33633,"thread_start_us":34063,"threads_started":1}
W20260812 20:01:07.710708 5837 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:07.639758 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:38719 (local address 127.25.14.62:35265)
0812 20:01:07.682217 (+ 42459us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:07.682226 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:07.682252 (+ 26us) server_negotiation.cc:408] Connection header received
0812 20:01:07.682334 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:07.682343 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:07.686431 (+ 4088us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:07.686565 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:07.708367 (+ 21802us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:38719: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":42282,"thread_start_us":50607,"threads_started":1}
W20260812 20:01:07.712242 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5368: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:35265: server requires authentication, but client does not have Kerberos credentials available
W20260812 20:01:07.712347 5837 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:07.710390 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.25.14.62:35265)
0812 20:01:07.711092 (+ 702us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:07.711099 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:07.711118 (+ 19us) server_negotiation.cc:408] Connection header received
0812 20:01:07.711188 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:07.711192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:07.711259 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:07.711387 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:07.712163 (+ 776us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33320: BlockingRecv error: recv got EOF from 127.0.0.1:33320 (error 108)
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":534,"threads_started":1}
I20260812 20:01:07.712450 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5368
I20260812 20:01:07.743960 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4556
I20260812 20:01:07.745322 4623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:07.865336 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4556
I20260812 20:01:07.896311 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4639
I20260812 20:01:07.897675 4711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:08.012324 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4639
I20260812 20:01:08.041500 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 4967
I20260812 20:01:08.042964 5166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:08.161902 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 4967
[ OK ] ExternalMiniClusterTest.TestAddMaster/6 (11071 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/7
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:01:08 dist-test-slave-p3t5 krb5kdc[5855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:01:08 dist-test-slave-p3t5 krb5kdc[5855](info): set up 2 sockets
Aug 12 20:01:08 dist-test-slave-p3t5 krb5kdc[5855](info): commencing operation
krb5kdc: starting...
W20260812 20:01:10.435906 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.229s user 0.009s sys 0.080s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:01:10 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:01:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:10.944839 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35891
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:11.343447 5873 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:11.343858 5873 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:11.343947 5873 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:11.352893 5873 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:11.353021 5873 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:11.353080 5873 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:11.353142 5873 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:11.364683 5873 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:11.366696 5873 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:11.368216 5873 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:11.377317 5878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:11.377316 5879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:11.378777 5881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:11.379068 5873 server_base.cc:1061] running on GCE node
I20260812 20:01:11.380064 5873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:11.381950 5873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:11.383399 5873 hybrid_clock.cc:648] HybridClock initialized: now 1786564871383158 us; error 81 us; skew 500 ppm
I20260812 20:01:11.383888 5873 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:11 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:11.387708 5873 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:11.389606 5873 webserver.cc:533] Webserver started at http://127.25.14.62:38161/ using document root <none> and password file <none>
I20260812 20:01:11.390486 5873 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:11.390592 5873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:11.390945 5873 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:11.393573 5873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "14b1002f93a542f7b85f6f7711222cbe"
format_stamp: "Formatted at 2026-08-12 20:01:11 on dist-test-slave-p3t5"
I20260812 20:01:11.394711 5873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "14b1002f93a542f7b85f6f7711222cbe"
format_stamp: "Formatted at 2026-08-12 20:01:11 on dist-test-slave-p3t5"
I20260812 20:01:11.401170 5873 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:01:11.405544 5889 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:11.407480 5873 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:11.407689 5873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14b1002f93a542f7b85f6f7711222cbe"
format_stamp: "Formatted at 2026-08-12 20:01:11 on dist-test-slave-p3t5"
I20260812 20:01:11.407889 5873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:11.432237 5873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:11.436545 5873 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:11.436826 5873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:11.460260 5873 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35891
I20260812 20:01:11.460279 5946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35891 every 8 connection(s)
I20260812 20:01:11.462112 5873 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:11.467074 5947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:11.468211 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5873
I20260812 20:01:11.468411 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:01:11.473541 5947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:11 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:11.500690 5947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:11.502254 5894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
W20260812 20:01:11.504118 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34319: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:11.553565 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } attempt: 1
W20260812 20:01:11.556275 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34319: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:11.624645 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } attempt: 2
W20260812 20:01:11.627111 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34319: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:11.641021 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:34319
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:11.718483 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } attempt: 3
W20260812 20:01:11.720821 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34319: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
I20260812 20:01:11.864250 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } attempt: 4
W20260812 20:01:11.866811 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:34319: Network error: Client connection negotiation failed: client connection to 127.25.14.61:34319: connect: Connection refused (error 111)
W20260812 20:01:12.036768 5958 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:12.037140 5958 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:12.037226 5958 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:12.046087 5958 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:12.046217 5958 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:12.046275 5958 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:12.046329 5958 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:12.057909 5958 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:12.059674 5958 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:12.061184 5958 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:12.070680 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:12.070724 5963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:12.071578 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:12.072096 5958 server_base.cc:1061] running on GCE node
I20260812 20:01:12.072979 5958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:12.074716 5958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:12.076010 5958 hybrid_clock.cc:648] HybridClock initialized: now 1786564872075898 us; error 66 us; skew 500 ppm
I20260812 20:01:12.076458 5958 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:12.080186 5958 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:01:12.082376 5958 webserver.cc:533] Webserver started at http://127.25.14.61:40505/ using document root <none> and password file <none>
I20260812 20:01:12.083242 5958 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:12.083398 5958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:12.083796 5958 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:12.086575 5958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "ffb29aef0be6466287ecdaeb12db7074"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.087514 5958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "ffb29aef0be6466287ecdaeb12db7074"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.094242 5958 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:01:12.098428 5975 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:12.100114 5958 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:01:12.100345 5958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "ffb29aef0be6466287ecdaeb12db7074"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.100574 5958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:12.120744 5958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:12.125568 5958 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:12.125958 5958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:12.150130 5958 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:34319
I20260812 20:01:12.150141 6030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:34319 every 8 connection(s)
I20260812 20:01:12.151954 5958 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:01:12.153952 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 5958
I20260812 20:01:12.154163 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:01:12.156875 6032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:12.163275 6032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:12.166134 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } attempt: 5
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:12.197446 5947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:12.198904 6032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:12.199801 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:12.215798 6032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:12.217100 5977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
W20260812 20:01:12.218947 6032 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
I20260812 20:01:12.251080 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:12.253419 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
I20260812 20:01:12.268400 6032 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 1
W20260812 20:01:12.270823 6032 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:01:12.321719 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 2
W20260812 20:01:12.324151 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:12.337750 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39691
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:12.339103 6032 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 2
W20260812 20:01:12.341357 6032 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
I20260812 20:01:12.430442 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 3
I20260812 20:01:12.432595 6032 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 3
W20260812 20:01:12.432761 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
W20260812 20:01:12.434828 6032 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
I20260812 20:01:12.578058 6032 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 4
W20260812 20:01:12.580585 6032 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
I20260812 20:01:12.609985 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 4
W20260812 20:01:12.612233 5947 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:39691: Network error: Client connection negotiation failed: client connection to 127.25.14.60:39691: connect: Connection refused (error 111)
W20260812 20:01:12.733836 6045 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:12.734233 6045 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:12.734387 6045 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:12.743181 6045 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:12.743314 6045 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:12.743460 6045 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:12.743544 6045 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:12.755111 6045 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:12.756932 6045 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:12.758536 6045 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:12.767737 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:12.767824 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:12.768586 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:12.769310 6045 server_base.cc:1061] running on GCE node
I20260812 20:01:12.770315 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:12.771886 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:12.773155 6045 hybrid_clock.cc:648] HybridClock initialized: now 1786564872773106 us; error 46 us; skew 500 ppm
I20260812 20:01:12.773576 6045 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:12.777244 6045 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:01:12.780238 6045 webserver.cc:533] Webserver started at http://127.25.14.60:36043/ using document root <none> and password file <none>
I20260812 20:01:12.781147 6045 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:12.781304 6045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:12.781767 6045 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:12.784493 6045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "4fe3e0dd89e64e3b8955c81e0025a19b"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.785344 6045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "4fe3e0dd89e64e3b8955c81e0025a19b"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.791757 6045 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:01:12.795884 6064 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:12.797703 6045 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:12.798030 6045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "4fe3e0dd89e64e3b8955c81e0025a19b"
format_stamp: "Formatted at 2026-08-12 20:01:12 on dist-test-slave-p3t5"
I20260812 20:01:12.798260 6045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:12.823822 6045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:12.828148 6045 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:12.828421 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:12.850744 6045 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:39691
I20260812 20:01:12.850781 6123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:39691 every 8 connection(s)
I20260812 20:01:12.852551 6045 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:01:12.857494 6124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:12.860517 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6045
I20260812 20:01:12.860749 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:01:12.866338 6124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:12.879879 6032 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 5
I20260812 20:01:12.889546 5947 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } attempt: 5
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:12.907884 6124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:12.916988 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: Bootstrap starting.
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:12.922411 6032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: Neither blocks nor log segments found. Creating new log.
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:12.924028 6032 log.cc:826] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:12.924072 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Bootstrap starting.
I20260812 20:01:12.928823 5947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:12.930534 5947 log.cc:826] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:12.935235 6124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:12.938432 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: No bootstrap required, opened a new log
I20260812 20:01:12.939224 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: No bootstrap required, opened a new log
Aug 12 20:01:12 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:12.946195 5947 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.946533 5947 raft_consensus.cc:385] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:12.946662 5947 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b1002f93a542f7b85f6f7711222cbe, State: Initialized, Role: FOLLOWER
I20260812 20:01:12.947542 5947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.947618 6032 raft_consensus.cc:359] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.947916 6032 raft_consensus.cc:385] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:12.948040 6032 raft_consensus.cc:740] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb29aef0be6466287ecdaeb12db7074, State: Initialized, Role: FOLLOWER
I20260812 20:01:12.948762 6032 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.952104 6137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:12.952649 6137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:12.953713 5947 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:12.954677 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:12.955107 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:12.956182 6032 sys_catalog.cc:565] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:12.960364 6124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Bootstrap starting.
I20260812 20:01:12.965222 6163 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 20:01:12.966356 6124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:12.967906 6124 log.cc:826] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:12.971994 6124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: No bootstrap required, opened a new log
W20260812 20:01:12.976801 6157 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:12.976962 6157 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:12.980258 6124 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.980718 6124 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:12.980854 6124 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe3e0dd89e64e3b8955c81e0025a19b, State: Initialized, Role: FOLLOWER
I20260812 20:01:12.981833 6124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:12.981422 5958 master_runner.cc:187] Error getting master registration for 127.25.14.60:39691: OK, instance_id { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" instance_seqno: 1786564872844670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:12.988063 5958 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:12.988912 6167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:12.989410 6167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:12.979461 5873 master_runner.cc:187] Error getting master registration for 127.25.14.60:39691: OK, instance_id { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" instance_seqno: 1786564872844670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:12.990267 5873 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:12.990622 6124 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:13.007808 6183 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:13.007994 6183 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:13.013360 6045 master_runner.cc:264] No leader master found from master 4fe3e0dd89e64e3b8955c81e0025a19b
I20260812 20:01:13.013558 6045 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 20:01:13.080869 6070 tablet.cc:2406] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:01:13.129082 5981 tablet.cc:2406] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:13.138414 6167 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:01:13.138808 6167 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:13.141235 6167 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b1002f93a542f7b85f6f7711222cbe (127.25.14.62:35891), ffb29aef0be6466287ecdaeb12db7074 (127.25.14.61:34319)
I20260812 20:01:13.142472 5919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b1002f93a542f7b85f6f7711222cbe" is_pre_election: true
I20260812 20:01:13.142544 6004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb29aef0be6466287ecdaeb12db7074" is_pre_election: true
I20260812 20:01:13.143133 5919 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe3e0dd89e64e3b8955c81e0025a19b in term 0.
I20260812 20:01:13.143124 6004 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe3e0dd89e64e3b8955c81e0025a19b in term 0.
I20260812 20:01:13.143955 6068 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe3e0dd89e64e3b8955c81e0025a19b, ffb29aef0be6466287ecdaeb12db7074; no voters:
I20260812 20:01:13.144595 6167 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:01:13.144893 6167 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:01:13.145057 6167 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:13.149196 6167 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:13.152153 6167 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [CANDIDATE]: Term 1 election: Requested vote from peers 14b1002f93a542f7b85f6f7711222cbe (127.25.14.62:35891), ffb29aef0be6466287ecdaeb12db7074 (127.25.14.61:34319)
I20260812 20:01:13.152812 6004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb29aef0be6466287ecdaeb12db7074"
I20260812 20:01:13.153044 6004 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:13.157177 6004 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe3e0dd89e64e3b8955c81e0025a19b in term 1.
I20260812 20:01:13.157876 6068 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe3e0dd89e64e3b8955c81e0025a19b, ffb29aef0be6466287ecdaeb12db7074; no voters:
I20260812 20:01:13.158455 6167 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:01:13.160185 5919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b1002f93a542f7b85f6f7711222cbe"
I20260812 20:01:13.160471 5919 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:13.164832 5919 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe3e0dd89e64e3b8955c81e0025a19b in term 1.
I20260812 20:01:13.170239 6167 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 1 LEADER]: Becoming Leader. State: Replica: 4fe3e0dd89e64e3b8955c81e0025a19b, State: Running, Role: LEADER
I20260812 20:01:13.170909 6167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } }
I20260812 20:01:13.178434 6185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe3e0dd89e64e3b8955c81e0025a19b. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.178824 6185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:13.180088 6187 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:01:13.182998 6187 catalog_manager.cc:1520] Initializing Kudu cluster ID...
W20260812 20:01:13.190920 5896 tablet.cc:2406] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:13.196228 6004 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:13.196383 5919 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:13.197552 6185 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:13.198020 6167 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:13.226317 6188 mvcc.cc:204] Tried to move back new op lower bound from 7317769720598220800 to 7317769720513544192. Current Snapshot: MvccSnapshot[applied={T|T < 7317769720598220800}]
I20260812 20:01:13.226555 6137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe3e0dd89e64e3b8955c81e0025a19b. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.226907 6137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:13.235775 6190 mvcc.cc:204] Tried to move back new op lower bound from 7317769720598220800 to 7317769720513544192. Current Snapshot: MvccSnapshot[applied={T|T < 7317769720598220800}]
I20260812 20:01:13.236438 6167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.236745 6167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:13.239952 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe3e0dd89e64e3b8955c81e0025a19b. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.240227 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:13.241113 6187 catalog_manager.cc:1383] Generated new cluster ID: 2159f3f1512e4da2b20b3293fef9e2c2
I20260812 20:01:13.241237 6187 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:01:13.246872 6137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.247138 6137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:13.257889 6189 mvcc.cc:204] Tried to move back new op lower bound from 7317769720598220800 to 7317769720513544192. Current Snapshot: MvccSnapshot[applied={T|T < 7317769720598220800}]
I20260812 20:01:13.262336 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.262631 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:13.280663 6196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } }
I20260812 20:01:13.282022 6196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:13.288667 6187 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:01:13.294296 6187 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:01:13.318078 6187 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Generated new TSK 0
I20260812 20:01:13.319872 6187 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:13.463464 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.117s user 0.002s sys 0.037s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:13.580379 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.117s user 0.005s sys 0.035s
I20260812 20:01:13.581445 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:13.971660 6163 catalog_manager.cc:1295] Loaded cluster ID: 2159f3f1512e4da2b20b3293fef9e2c2
I20260812 20:01:13.971812 6163 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: loading cluster ID for follower catalog manager: success
I20260812 20:01:13.976114 6163 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: acquiring CA information for follower catalog manager: success
I20260812 20:01:13.979241 6163 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:13.979416 6157 catalog_manager.cc:1295] Loaded cluster ID: 2159f3f1512e4da2b20b3293fef9e2c2
I20260812 20:01:13.979530 6157 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: loading cluster ID for follower catalog manager: success
I20260812 20:01:13.983507 6157 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: acquiring CA information for follower catalog manager: success
I20260812 20:01:13.986300 6157 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 20:01:14.026194 6203 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:14.026597 6203 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:14.026770 6203 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:14.036607 6203 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:14.036909 6203 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:14.049552 6203 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:14.051563 6203 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:14.053267 6203 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:14.065758 6211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:14.065797 6210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:14.069895 6203 server_base.cc:1061] running on GCE node
W20260812 20:01:14.070925 6213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:14.071841 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:14.073071 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:14.074337 6203 hybrid_clock.cc:648] HybridClock initialized: now 1786564874074261 us; error 66 us; skew 500 ppm
I20260812 20:01:14.074712 6203 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:14.078583 6203 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:14.080742 6203 webserver.cc:533] Webserver started at http://127.25.14.1:38137/ using document root <none> and password file <none>
I20260812 20:01:14.081682 6203 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:14.081845 6203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:14.082270 6203 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:14.085460 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.086500 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.093364 6203 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260812 20:01:14.097939 6220 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:14.099681 6203 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:01:14.099880 6203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.100102 6203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:14.118539 6203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:14.128054 6203 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:14.128437 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:14.129787 6203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:14.131822 6203 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:14.131947 6203 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:14.132115 6203 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:14.132189 6203 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:14.184145 6203 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45661
I20260812 20:01:14.184181 6348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45661 every 8 connection(s)
I20260812 20:01:14.186121 6203 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:14.195542 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6203
I20260812 20:01:14.195760 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:14.241878 6349 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39691
I20260812 20:01:14.242398 6349 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:14.243729 6349 heartbeater.cc:507] Master 127.25.14.60:39691 requested a full tablet report, sending...
I20260812 20:01:14.246762 6083 ts_manager.cc:194] Registered new tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:14.249248 6083 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:45235
I20260812 20:01:14.249419 6350 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35891
I20260812 20:01:14.249576 6350 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:14.250170 6350 heartbeater.cc:507] Master 127.25.14.62:35891 requested a full tablet report, sending...
I20260812 20:01:14.252679 5908 ts_manager.cc:194] Registered new tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:14.253862 6351 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34319
I20260812 20:01:14.254034 6351 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:14.254452 6351 heartbeater.cc:507] Master 127.25.14.61:34319 requested a full tablet report, sending...
I20260812 20:01:14.256484 5992 ts_manager.cc:194] Registered new tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:14.259997 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:14.364965 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.59: real 0.103s user 0.007s sys 0.028s
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:14.455389 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:41797
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:14.846606 6363 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:14.846980 6363 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:14.847075 6363 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:14.855878 6363 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:14.855993 6363 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:14.856107 6363 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:14.856189 6363 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:14.869035 6363 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:41797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:14.870957 6363 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:14.872496 6363 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:14.881808 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:14.881810 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:14.881860 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:14.883265 6363 server_base.cc:1061] running on GCE node
I20260812 20:01:14.884225 6363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:14.885892 6363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:14.887141 6363 hybrid_clock.cc:648] HybridClock initialized: now 1786564874887095 us; error 37 us; skew 500 ppm
I20260812 20:01:14.887545 6363 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:14.891408 6363 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:14.893431 6363 webserver.cc:533] Webserver started at http://127.25.14.59:40135/ using document root <none> and password file <none>
I20260812 20:01:14.894413 6363 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:14.894572 6363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:14.894982 6363 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:14.897900 6363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.898811 6363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.905315 6363 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:01:14.909421 6381 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:14.911034 6363 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 20:01:14.911266 6363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:14.911500 6363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:14.925318 6363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:14.929837 6363 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:14.930152 6363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:14.952180 6363 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:41797
I20260812 20:01:14.952200 6438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:41797 every 8 connection(s)
I20260812 20:01:14.954073 6363 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:14.957370 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6363
I20260812 20:01:14.960748 6439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:14.967465 6439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:14.986500 6439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:14 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:14.999778 6439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:15.013571 6439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:15.031147 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Bootstrap starting.
I20260812 20:01:15.034552 6439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:15.035925 6439 log.cc:826] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:15.038893 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: No bootstrap required, opened a new log
I20260812 20:01:15.044324 6439 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } }
I20260812 20:01:15.044711 6439 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:15.044827 6439 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8f5678b62141a49cf87dd8bcc1679c, State: Initialized, Role: FOLLOWER
I20260812 20:01:15.045518 6439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } }
I20260812 20:01:15.046655 6449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } } }
I20260812 20:01:15.047010 6449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:15.047792 6439 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:15.058902 6462 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:15.059072 6462 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:15.067256 6363 master_runner.cc:429] Detected that this master 4c8f5678b62141a49cf87dd8bcc1679c is joining an existing cluster
I20260812 20:01:15.067427 6363 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:01:15.068475 6363 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:01:15.068858 6363 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:01:15.069077 6363 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: stopping tablet replica
I20260812 20:01:15.070822 6363 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:41797
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:15.136797 6083 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:41797: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 41797
}
attrs {
promote: true
}
}
dest_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b"
cas_config_opid_index: -1
I20260812 20:01:15.138607 6083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } }
I20260812 20:01:15.142076 5918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:01:15.142868 6196 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260812 20:01:15.143280 6065 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b -> Peer 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59:41797): Couldn't send request to peer 4c8f5678b62141a49cf87dd8bcc1679c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:15.144917 6004 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:01:15.145556 6196 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:15.146845 6475 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.147526 5918 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.158645 6478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.159955 6478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:15.164661 6004 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.165776 6475 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:41797
I20260812 20:01:15.166724 6363 master.cc:562] Master@127.25.14.59:41797 shutting down...
I20260812 20:01:15.175653 6196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.175990 6196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:15.181929 6475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.182225 6475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:15.182519 6475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.187626 6475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:15.195108 6363 master.cc:584] Master@127.25.14.59:41797 shutdown complete.
I20260812 20:01:15.195225 6363 master_runner.cc:306] Clearing existing system tablet
I20260812 20:01:15.195714 6363 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:15.195837 6363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:15.199702 6480 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.200132 6480 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:15.202421 6363 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 20:01:15.204329 6483 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:15.204747 6363 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:15.205035 6363 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 20:01:15.205142 6363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:15.210248 6363 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:01:15.217819 6363 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:01:15.218061 6363 log.cc:1199] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:01:15.218573 6363 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Deleting consensus metadata
I20260812 20:01:15.218873 6363 master_runner.cc:316] Copying system tablet from 127.25.14.60:39691
I20260812 20:01:15.221936 6363 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:39691
I20260812 20:01:15.236858 6106 tablet_copy_service.cc:140] P 4fe3e0dd89e64e3b8955c81e0025a19b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4c8f5678b62141a49cf87dd8bcc1679c ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54500)
I20260812 20:01:15.237202 6106 tablet_copy_service.cc:161] P 4fe3e0dd89e64e3b8955c81e0025a19b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4c8f5678b62141a49cf87dd8bcc1679c at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54500: session id = 4c8f5678b62141a49cf87dd8bcc1679c-00000000000000000000000000000000
I20260812 20:01:15.240263 6106 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:01:15.242779 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:01:15.250460 6363 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: tablet copy: Starting download of 0 data blocks...
I20260812 20:01:15.250828 6363 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: tablet copy: Starting download of 1 WAL segments...
I20260812 20:01:15.253914 6363 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:01:15.254508 6349 heartbeater.cc:499] Master 127.25.14.60:39691 was elected leader, sending a full tablet report...
I20260812 20:01:15.258848 6106 tablet_copy_service.cc:342] P 4fe3e0dd89e64e3b8955c81e0025a19b: Request end of tablet copy session 4c8f5678b62141a49cf87dd8bcc1679c-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54500
I20260812 20:01:15.259171 6106 tablet_copy_service.cc:434] P 4fe3e0dd89e64e3b8955c81e0025a19b: ending tablet copy session 4c8f5678b62141a49cf87dd8bcc1679c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4c8f5678b62141a49cf87dd8bcc1679c
W20260812 20:01:15.266788 6363 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:01:15.271277 6363 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:15.274134 6492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:15.274262 6491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:15.275413 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:15.275627 6363 server_base.cc:1061] running on GCE node
I20260812 20:01:15.276065 6363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:15.276727 6363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:15.277936 6363 hybrid_clock.cc:648] HybridClock initialized: now 1786564875277890 us; error 37 us; skew 500 ppm
I20260812 20:01:15.278204 6363 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:15.280537 6363 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:15.281785 6363 webserver.cc:533] Webserver started at http://127.25.14.59:44765/ using document root <none> and password file <none>
I20260812 20:01:15.282145 6363 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:15.282258 6363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:15.284615 6363 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260812 20:01:15.286501 6502 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:15.286909 6363 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 20:01:15.287052 6363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "4c8f5678b62141a49cf87dd8bcc1679c"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:15.287228 6363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:15.295797 6363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:15.299505 6363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:15.319414 6363 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:41797
I20260812 20:01:15.319523 6560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:41797 every 8 connection(s)
I20260812 20:01:15.320514 6363 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:15.325498 6561 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:15.329209 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Bootstrap starting.
I20260812 20:01:15.348294 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:15.348889 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: Bootstrap complete.
I20260812 20:01:15.350317 6561 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } }
I20260812 20:01:15.350613 6561 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c8f5678b62141a49cf87dd8bcc1679c, State: Initialized, Role: LEARNER
I20260812 20:01:15.350886 6561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } }
I20260812 20:01:15.351832 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.352084 6561 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:15.352327 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:15.362264 6580 catalog_manager.cc:1295] Loaded cluster ID: 2159f3f1512e4da2b20b3293fef9e2c2
I20260812 20:01:15.362399 6580 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: loading cluster ID for follower catalog manager: success
I20260812 20:01:15.366531 6580 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: acquiring CA information for follower catalog manager: success
I20260812 20:01:15.367216 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6203
I20260812 20:01:15.369442 6580 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:01:15.387928 6343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:15.503628 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6203
I20260812 20:01:15.530512 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45661
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--webserver_port=38137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:15.540050 6532 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:01:15.541064 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe3e0dd89e64e3b8955c81e0025a19b. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:15.541371 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: This master's current role is: LEARNER
W20260812 20:01:15.948745 6583 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:15.949365 6583 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:15.949529 6583 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:15.958690 6583 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:15.958904 6583 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:15.970297 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:15.972030 6583 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:15.973452 6583 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:15.983975 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:15.984005 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:15.984396 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:15.985194 6583 server_base.cc:1061] running on GCE node
I20260812 20:01:15.986120 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:15.987284 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:15.988547 6583 hybrid_clock.cc:648] HybridClock initialized: now 1786564875988489 us; error 48 us; skew 500 ppm
I20260812 20:01:15.988878 6583 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:15 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:15.997468 6583 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:15.999500 6583 webserver.cc:533] Webserver started at http://127.25.14.1:38137/ using document root <none> and password file <none>
I20260812 20:01:16.000319 6583 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:16.000445 6583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:16.006704 6583 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:16.010080 6601 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:16.011744 6583 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:01:16.011955 6583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:16.012840 6583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:16.031826 6583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:16.036072 6583 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:16.036389 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:16.037523 6583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:16.039419 6583 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:16.039521 6583 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:16.039618 6583 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:16.039673 6583 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:16.077807 6583 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45661
I20260812 20:01:16.077868 6736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45661 every 8 connection(s)
I20260812 20:01:16.079492 6583 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:16.081522 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6583
Aug 12 20:01:16 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:16 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:16 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:16 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:16.128243 6737 heartbeater.cc:344] Connected to a master server at 127.25.14.59:41797
I20260812 20:01:16.128628 6737 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:16.129904 6737 heartbeater.cc:507] Master 127.25.14.59:41797 requested a full tablet report, sending...
I20260812 20:01:16.130434 6739 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35891
I20260812 20:01:16.130656 6739 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:16.131078 6739 heartbeater.cc:507] Master 127.25.14.62:35891 requested a full tablet report, sending...
I20260812 20:01:16.132630 5908 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:16.132841 6521 ts_manager.cc:194] Registered new tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:16.133661 6740 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34319
I20260812 20:01:16.133823 6740 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:16.134335 6740 heartbeater.cc:507] Master 127.25.14.61:34319 requested a full tablet report, sending...
I20260812 20:01:16.135526 6738 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39691
I20260812 20:01:16.135672 6738 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:16.135828 5992 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:16.136058 6738 heartbeater.cc:507] Master 127.25.14.60:39691 requested a full tablet report, sending...
I20260812 20:01:16.137228 6083 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:16.137775 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:16.138444 6083 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:59025
I20260812 20:01:16.139343 6479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:16.139608 6479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:16.142207 6479 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b: attempting to promote NON_VOTER 4c8f5678b62141a49cf87dd8bcc1679c to VOTER
I20260812 20:01:16.143211 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } }
I20260812 20:01:16.145809 6004 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:16.145954 6532 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 1 LEARNER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:16.146694 6750 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:16.147089 5918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Refusing update from remote peer 4fe3e0dd89e64e3b8955c81e0025a19b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:16.147234 6754 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:16.148507 6750 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:16.148718 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:16.148885 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5873
I20260812 20:01:16.149884 5941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:16.151948 6754 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.157406 6532 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.163111 6759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.163781 6759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8f5678b62141a49cf87dd8bcc1679c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:16.166568 6479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.166844 6479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:16.167191 6479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.167373 6479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:16.167618 6479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.167773 6479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:16.168102 6004 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.171934 6760 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:16.172477 6760 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb29aef0be6466287ecdaeb12db7074 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:16.311931 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5873
W20260812 20:01:16.351373 6066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.14.62:35891 (error 108)
I20260812 20:01:16.352419 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35891
--webserver_interface=127.25.14.62
--webserver_port=38161
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:16.354149 6066 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b -> Peer 14b1002f93a542f7b85f6f7711222cbe (127.25.14.62:35891): Couldn't send request to peer 14b1002f93a542f7b85f6f7711222cbe. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:01:16.751725 6763 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:16.752148 6763 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:16.752312 6763 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:16.761423 6763 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:16.761546 6763 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:16.761658 6763 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:16.761739 6763 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:16.773550 6763 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=38161
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:16.775434 6763 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:16.777040 6763 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:16.786665 6768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:16.786695 6769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:16.787645 6772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:16.788218 6763 server_base.cc:1061] running on GCE node
I20260812 20:01:16.789160 6763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:16.790997 6763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:16.792261 6763 hybrid_clock.cc:648] HybridClock initialized: now 1786564876792212 us; error 74 us; skew 500 ppm
I20260812 20:01:16.792701 6763 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:16 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564876, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:16.796799 6763 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:16.799088 6763 webserver.cc:533] Webserver started at http://127.25.14.62:38161/ using document root <none> and password file <none>
I20260812 20:01:16.800117 6763 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:16.800288 6763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:16.807353 6763 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 20:01:16.810828 6779 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:16.812386 6763 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:01:16.812601 6763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "14b1002f93a542f7b85f6f7711222cbe"
format_stamp: "Formatted at 2026-08-12 20:01:11 on dist-test-slave-p3t5"
I20260812 20:01:16.813452 6763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:16.843421 6763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:16.848081 6763 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:16.848414 6763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:16.872593 6763 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35891
I20260812 20:01:16.872601 6838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35891 every 8 connection(s)
I20260812 20:01:16.874644 6763 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:16.883018 6839 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:16.884876 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6763
I20260812 20:01:16.885207 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6583
I20260812 20:01:16.887718 6839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Bootstrap starting.
I20260812 20:01:16.906327 6732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:16.912035 6839 log.cc:826] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:16.923395 6839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:16.924021 6839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: Bootstrap complete.
I20260812 20:01:16.931507 6839 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } }
I20260812 20:01:16.932348 6839 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b1002f93a542f7b85f6f7711222cbe, State: Initialized, Role: FOLLOWER
I20260812 20:01:16.933177 6839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } }
I20260812 20:01:16.934203 6846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: true } } }
I20260812 20:01:16.934559 6846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:16.935492 6839 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:16.956418 6858 catalog_manager.cc:1295] Loaded cluster ID: 2159f3f1512e4da2b20b3293fef9e2c2
I20260812 20:01:16.956954 6858 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: loading cluster ID for follower catalog manager: success
I20260812 20:01:16.976610 6858 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: acquiring CA information for follower catalog manager: success
I20260812 20:01:16.981244 6858 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:17.012039 6810 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4c8f5678b62141a49cf87dd8bcc1679c (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:17.029021 6846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe3e0dd89e64e3b8955c81e0025a19b. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:17.029356 6846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:17.028293 6866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:17.032064 6866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b1002f93a542f7b85f6f7711222cbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:17.043702 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6583
I20260812 20:01:17.073479 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45661
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--webserver_port=38137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:17.432811 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14b1002f93a542f7b85f6f7711222cbe" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35891 } } peers { permanent_uuid: "ffb29aef0be6466287ecdaeb12db7074" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 34319 } } peers { permanent_uuid: "4fe3e0dd89e64e3b8955c81e0025a19b" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 39691 } } peers { permanent_uuid: "4c8f5678b62141a49cf87dd8bcc1679c" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 41797 } attrs { promote: false } } }
I20260812 20:01:17.433241 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b [sys.catalog]: This master's current role is: LEADER
W20260812 20:01:17.487780 6868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:17.488135 6868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:17.488234 6868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:17.497225 6868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:17.497432 6868 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:17.508774 6868 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:17.510609 6868 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:17.512059 6868 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:17.522871 6878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:17.523231 6877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:17.523833 6880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:17.527773 6868 server_base.cc:1061] running on GCE node
I20260812 20:01:17.528527 6868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:17.529672 6868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:17.530936 6868 hybrid_clock.cc:648] HybridClock initialized: now 1786564877530871 us; error 47 us; skew 500 ppm
I20260812 20:01:17.531286 6868 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:17 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:17.535024 6868 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:17.537259 6868 webserver.cc:533] Webserver started at http://127.25.14.1:38137/ using document root <none> and password file <none>
I20260812 20:01:17.538174 6868 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:17.538321 6868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:17.544461 6868 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260812 20:01:17.547720 6888 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:17.549410 6868 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:01:17.549685 6868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:17.550493 6868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:17.570366 6868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:17.574975 6868 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:17.575299 6868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:17.576411 6868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:17.578419 6868 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:17.578549 6868 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:17.578701 6868 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:17.578810 6868 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:17.616191 6868 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45661
I20260812 20:01:17.616289 7019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45661 every 8 connection(s)
I20260812 20:01:17.617971 6868 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:17.628448 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 6868
Aug 12 20:01:17 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:17 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:17 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:17.679195 7023 heartbeater.cc:344] Connected to a master server at 127.25.14.61:34319
I20260812 20:01:17.679692 7023 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:17.680404 7020 heartbeater.cc:344] Connected to a master server at 127.25.14.59:41797
I20260812 20:01:17.680666 7020 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:17.681063 7023 heartbeater.cc:507] Master 127.25.14.61:34319 requested a full tablet report, sending...
I20260812 20:01:17.681185 7020 heartbeater.cc:507] Master 127.25.14.59:41797 requested a full tablet report, sending...
I20260812 20:01:17.683635 5992 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
Aug 12 20:01:17 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:17.684185 6521 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:17.685443 7021 heartbeater.cc:344] Connected to a master server at 127.25.14.60:39691
I20260812 20:01:17.685609 7021 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:17.686280 7021 heartbeater.cc:507] Master 127.25.14.60:39691 requested a full tablet report, sending...
I20260812 20:01:17.687587 6083 ts_manager.cc:194] Re-registered known tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:17.688730 6083 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:41405
I20260812 20:01:17.694327 7022 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35891
I20260812 20:01:17.694522 7022 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:17.694885 7022 heartbeater.cc:507] Master 127.25.14.62:35891 requested a full tablet report, sending...
I20260812 20:01:17.696671 6799 ts_manager.cc:194] Registered new tserver with Master: 4dab9f53f66044e584b4e43b114eaefe (127.25.14.1:45661)
I20260812 20:01:17.725240 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6868
W20260812 20:01:17.726644 7036 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:17.722967 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.25.14.1:45661)
0812 20:01:17.723598 (+ 631us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:17.723607 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:17.723669 (+ 62us) server_negotiation.cc:408] Connection header received
0812 20:01:17.723740 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:17.723747 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:17.723989 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:17.724163 (+ 174us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:17.725070 (+ 907us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34430: BlockingRecv error: recv got EOF from 127.0.0.1:34430 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
W20260812 20:01:17.728055 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6868: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:45661: server requires authentication, but client does not have Kerberos credentials available
W20260812 20:01:17.728135 7036 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:17.726266 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.25.14.1:45661)
0812 20:01:17.727002 (+ 736us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:17.727011 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:17.727026 (+ 15us) server_negotiation.cc:408] Connection header received
0812 20:01:17.727078 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:17.727082 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:17.727142 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:17.727254 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:17.727954 (+ 700us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34446: BlockingRecv error: recv got EOF from 127.0.0.1:34446 (error 108)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":473,"threads_started":1}
I20260812 20:01:17.728193 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6868
I20260812 20:01:17.754619 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45661
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--webserver_port=38137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:18.163470 7039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:18.163852 7039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:18.164023 7039 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:01:18.164093 7039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:18.172713 7039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:18.172976 7039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:18.184657 7039 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:18.186793 7039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:18.188548 7039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:18.200204 7047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:18.201153 7046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:18.205793 7049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:18.206418 7039 server_base.cc:1061] running on GCE node
I20260812 20:01:18.207773 7039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:18.209214 7039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:18.210553 7039 hybrid_clock.cc:648] HybridClock initialized: now 1786564878210456 us; error 62 us; skew 500 ppm
I20260812 20:01:18.210999 7039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:18 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:18.214862 7039 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:18.216688 7039 webserver.cc:533] Webserver started at http://127.25.14.1:38137/ using document root <none> and password file <none>
I20260812 20:01:18.217551 7039 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:18.217717 7039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:18.224046 7039 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 20:01:18.227385 7057 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:18.229094 7039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:18.229280 7039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:18.230120 7039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:18.244902 7039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:18.249001 7039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:18.249327 7039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:18.250499 7039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:18.252305 7039 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:18.252424 7039 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:18.252583 7039 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:18.252656 7039 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:01:18.253095 7039 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:01:18.253180 7039 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:01:18.253263 7039 fault_injection.cc:59] injecting fault for kudu (pid 7039): FLAGS_fault_before_start (process will exit)
I20260812 20:01:18.278214 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:45661
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691
--webserver_port=38137
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:18.671411 7066 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:18.671775 7066 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:18.671984 7066 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:18.680800 7066 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:18.681084 7066 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:18.692945 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=38137
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35891,127.25.14.61:34319,127.25.14.60:39691,127.25.14.59:41797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:18.694885 7066 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:18.696408 7066 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:18.707677 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:18.707680 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:18.707940 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:18.709141 7066 server_base.cc:1061] running on GCE node
I20260812 20:01:18.709852 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:18.710870 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:18.712116 7066 hybrid_clock.cc:648] HybridClock initialized: now 1786564878712048 us; error 62 us; skew 500 ppm
I20260812 20:01:18.712456 7066 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:18 dist-test-slave-p3t5 krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:18.716109 7066 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:18.717907 7066 webserver.cc:533] Webserver started at http://127.25.14.1:38137/ using document root <none> and password file <none>
I20260812 20:01:18.718719 7066 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:18.718824 7066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:18.725142 7066 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 20:01:18.728430 7083 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:18.730115 7066 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 20:01:18.730341 7066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4dab9f53f66044e584b4e43b114eaefe"
format_stamp: "Formatted at 2026-08-12 20:01:14 on dist-test-slave-p3t5"
I20260812 20:01:18.731169 7066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:18.751793 7066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:18.756865 7066 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:18.757186 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:18.758352 7066 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:18.760197 7066 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:18.760329 7066 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:18.760475 7066 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:18.760565 7066 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:18.799456 7066 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:45661
I20260812 20:01:18.799614 7215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:45661 every 8 connection(s)
I20260812 20:01:18.801270 7066 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:18.810222 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7066
I20260812 20:01:18.810617 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7066
Aug 12 20:01:18 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:18 dist-test-slave-p3t5 krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
W20260812 20:01:18.843794 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7066: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:45661: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:18.843961 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7066
I20260812 20:01:18.870625 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6763
W20260812 20:01:18.871284 7225 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:18.808229 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:33177 (local address 127.25.14.59:41797)
0812 20:01:18.827660 (+ 19431us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:18.827666 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:18.827686 (+ 20us) server_negotiation.cc:408] Connection header received
0812 20:01:18.827749 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:18.827755 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:18.827839 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:18.829749 (+ 1910us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:18.829993 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.832269 (+ 2276us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:18.836208 (+ 3939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.838278 (+ 2070us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:18.839133 (+ 855us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:01:18.839168 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:01:18.839173 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 20:01:18.839211 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:01:18.848803 (+ 9592us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:18.870234 (+ 21431us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:33177: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19283,"thread_start_us":19164,"threads_started":1}
W20260812 20:01:18.871261 7222 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:18.805432 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:34113 (local address 127.25.14.60:39691)
0812 20:01:18.809142 (+ 3710us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:18.809152 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:18.820410 (+ 11258us) server_negotiation.cc:408] Connection header received
0812 20:01:18.823080 (+ 2670us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:18.823089 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:18.823196 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:18.823337 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:18.827230 (+ 3893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.828751 (+ 1521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:18.831197 (+ 2446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.831698 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:18.839043 (+ 7345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:01:18.839091 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:01:18.839096 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 20:01:18.839141 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:01:18.843507 (+ 4366us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:18.870236 (+ 26729us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:34113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":3532,"thread_start_us":3379,"threads_started":1}
W20260812 20:01:18.871264 7229 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:18.818945 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:56775 (local address 127.25.14.62:35891)
0812 20:01:18.833717 (+ 14772us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:18.833724 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:18.834606 (+ 882us) server_negotiation.cc:408] Connection header received
0812 20:01:18.834704 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:18.834714 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:18.834845 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:18.834974 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:18.839413 (+ 4439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.841145 (+ 1732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:18.848191 (+ 7046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.848628 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:18.870113 (+ 21485us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:56775: BlockingRecv error: recv got EOF from 127.25.14.1:56775 (error 108)
Metrics: {"server-negotiator.queue_time_us":14592,"thread_start_us":14974,"threads_started":1}
W20260812 20:01:18.873468 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6763: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:35891: server requires authentication, but client does not have Kerberos credentials available
W20260812 20:01:18.873589 7229 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:18.871936 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.25.14.62:35891)
0812 20:01:18.872113 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:18.872140 (+ 27us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:18.872160 (+ 20us) server_negotiation.cc:408] Connection header received
0812 20:01:18.872237 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:18.872249 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:18.872338 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:18.872458 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:18.873342 (+ 884us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50324: BlockingRecv error: recv got EOF from 127.0.0.1:50324 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 20:01:18.873663 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6763
W20260812 20:01:18.876911 7221 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:18.805301 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:41115 (local address 127.25.14.61:34319)
0812 20:01:18.832786 (+ 27485us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:18.832792 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:18.832815 (+ 23us) server_negotiation.cc:408] Connection header received
0812 20:01:18.832883 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:18.832889 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:18.832986 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:18.833121 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:18.838541 (+ 5420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:18.842525 (+ 3984us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:18.872706 (+ 30181us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:41115: BlockingRecv error: recv got EOF from 127.25.14.1:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":27352,"thread_start_us":27239,"threads_started":1}
I20260812 20:01:18.904408 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 5958
I20260812 20:01:18.905936 6026 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 20:01:18.948630 6066 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4fe3e0dd89e64e3b8955c81e0025a19b -> Peer 14b1002f93a542f7b85f6f7711222cbe (127.25.14.62:35891): Couldn't send request to peer 14b1002f93a542f7b85f6f7711222cbe. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:35891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:19.022459 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 5958
I20260812 20:01:19.052644 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6045
I20260812 20:01:19.054122 6118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:19.173151 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6045
W20260812 20:01:19.203280 6506 connection.cc:573] server connection from 127.0.0.1:38994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:01:19.203737 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 6363
I20260812 20:01:19.205394 6556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:19.324839 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 6363
2026-08-12T20:01:19Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/7 (11169 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/8
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:01:19 dist-test-slave-p3t5 krb5kdc[7242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:01:19 dist-test-slave-p3t5 krb5kdc[7242](info): set up 2 sockets
Aug 12 20:01:19 dist-test-slave-p3t5 krb5kdc[7242](info): commencing operation
krb5kdc: starting...
W20260812 20:01:21.580056 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.203s user 0.000s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:01:21 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:01:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:21Z Disabled control of system clock
2026-08-12T20:01:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:21Z Disabled control of system clock
2026-08-12T20:01:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:22Z Disabled control of system clock
2026-08-12T20:01:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:22Z Disabled control of system clock
2026-08-12T20:01:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:22.334113 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36151
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:22.723021 7272 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:22.723424 7272 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:22.723614 7272 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:22.732474 7272 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:22.732594 7272 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:22.732695 7272 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:22.732767 7272 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:22.744306 7272 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:22.746269 7272 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:22.747943 7272 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:22.757218 7278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:22.757360 7277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:22.757778 7280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:22.758167 7272 server_base.cc:1061] running on GCE node
I20260812 20:01:22.759027 7272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:22.760815 7272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:22.762161 7272 hybrid_clock.cc:648] HybridClock initialized: now 1786564882762013 us; error 111 us; skew 500 ppm
I20260812 20:01:22.762553 7272 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:22 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564882, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:22.766083 7272 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:22.767923 7272 webserver.cc:533] Webserver started at http://127.25.14.62:40453/ using document root <none> and password file <none>
I20260812 20:01:22.768725 7272 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:22.768832 7272 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:22.769183 7272 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:22.771881 7272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "0692a5f31f6042a0ad3df7e96143abd3"
format_stamp: "Formatted at 2026-08-12 20:01:22 on dist-test-slave-p3t5"
I20260812 20:01:22.772769 7272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "0692a5f31f6042a0ad3df7e96143abd3"
format_stamp: "Formatted at 2026-08-12 20:01:22 on dist-test-slave-p3t5"
I20260812 20:01:22.778867 7272 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:01:22.782847 7289 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:22.784507 7272 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:01:22.784732 7272 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "0692a5f31f6042a0ad3df7e96143abd3"
format_stamp: "Formatted at 2026-08-12 20:01:22 on dist-test-slave-p3t5"
I20260812 20:01:22.784971 7272 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:22.799185 7272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:22.803570 7272 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:22.803910 7272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:22.826642 7272 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:36151
I20260812 20:01:22.826664 7342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:36151 every 8 connection(s)
I20260812 20:01:22.828444 7272 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:22.833191 7343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:22.835014 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7272
I20260812 20:01:22.835224 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:01:22.839761 7343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:22 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564882, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:22.866057 7343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:22.867600 7293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
W20260812 20:01:22.869451 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40801: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:22.918874 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } attempt: 1
W20260812 20:01:22.921231 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40801: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:22.989552 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } attempt: 2
W20260812 20:01:22.992144 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40801: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:23.005681 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40801
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:23.083477 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } attempt: 3
W20260812 20:01:23.085932 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40801: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
I20260812 20:01:23.229264 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } attempt: 4
W20260812 20:01:23.231829 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:40801: Network error: Client connection negotiation failed: client connection to 127.25.14.61:40801: connect: Connection refused (error 111)
W20260812 20:01:23.398973 7354 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:23.399345 7354 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:23.399432 7354 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:23.408262 7354 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:23.408392 7354 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:23.408454 7354 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:23.408497 7354 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:23.420017 7354 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:23.421818 7354 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:23.423314 7354 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:23.432260 7360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:23.432497 7361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:23.432822 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:23.433706 7354 server_base.cc:1061] running on GCE node
I20260812 20:01:23.434589 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:23.436444 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:23.437733 7354 hybrid_clock.cc:648] HybridClock initialized: now 1786564883437564 us; error 131 us; skew 500 ppm
I20260812 20:01:23.438138 7354 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:23 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564883, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:23.442095 7354 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:01:23.443928 7354 webserver.cc:533] Webserver started at http://127.25.14.61:34997/ using document root <none> and password file <none>
I20260812 20:01:23.444794 7354 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:23.444934 7354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:23.445345 7354 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:23.448338 7354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "55baa567a6b24e18aa2aba6888d0f714"
format_stamp: "Formatted at 2026-08-12 20:01:23 on dist-test-slave-p3t5"
I20260812 20:01:23.449255 7354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "55baa567a6b24e18aa2aba6888d0f714"
format_stamp: "Formatted at 2026-08-12 20:01:23 on dist-test-slave-p3t5"
I20260812 20:01:23.455474 7354 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 20:01:23.459364 7370 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:23.460996 7354 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 20:01:23.461196 7354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "55baa567a6b24e18aa2aba6888d0f714"
format_stamp: "Formatted at 2026-08-12 20:01:23 on dist-test-slave-p3t5"
I20260812 20:01:23.461448 7354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:23.476339 7354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:23.480494 7354 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:23.480831 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:23.504027 7354 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:40801
I20260812 20:01:23.504065 7430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:40801 every 8 connection(s)
I20260812 20:01:23.505920 7354 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:01:23.507752 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7354
I20260812 20:01:23.507982 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:01:23.512276 7431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:23.522012 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:23.531111 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } attempt: 5
Aug 12 20:01:23 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564883, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:23 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564882, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:23.549466 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
I20260812 20:01:23.553112 7343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
W20260812 20:01:23.555327 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
Aug 12 20:01:23 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564883, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:23.564988 7431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:23.566417 7373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
W20260812 20:01:23.569453 7431 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:23.606614 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 1
W20260812 20:01:23.608980 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
I20260812 20:01:23.618852 7431 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 1
W20260812 20:01:23.621176 7431 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:01:23.677237 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 2
W20260812 20:01:23.679440 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:23.681715 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:34851
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:23.689459 7431 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 2
W20260812 20:01:23.691663 7431 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
I20260812 20:01:23.782902 7431 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 3
W20260812 20:01:23.785349 7431 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
I20260812 20:01:23.785748 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 3
W20260812 20:01:23.787930 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
I20260812 20:01:23.928637 7431 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 4
W20260812 20:01:23.931051 7431 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
I20260812 20:01:23.965237 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 4
W20260812 20:01:23.967774 7343 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:34851: Network error: Client connection negotiation failed: client connection to 127.25.14.60:34851: connect: Connection refused (error 111)
W20260812 20:01:24.068974 7442 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:24.069406 7442 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:24.069563 7442 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:24.078330 7442 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:24.078500 7442 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:24.078629 7442 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:24.078713 7442 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:24.090209 7442 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:24.092057 7442 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:24.093598 7442 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:24.102754 7447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:24.102788 7448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:24.103166 7450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:24.104033 7442 server_base.cc:1061] running on GCE node
I20260812 20:01:24.104961 7442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:24.106817 7442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:24.108119 7442 hybrid_clock.cc:648] HybridClock initialized: now 1786564884107963 us; error 121 us; skew 500 ppm
I20260812 20:01:24.108546 7442 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564884, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:24.111981 7442 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:01:24.113802 7442 webserver.cc:533] Webserver started at http://127.25.14.60:44553/ using document root <none> and password file <none>
I20260812 20:01:24.114604 7442 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:24.114750 7442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:24.115125 7442 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:24.117859 7442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "2a7d3d01abea41b89bf31e9e46051f99"
format_stamp: "Formatted at 2026-08-12 20:01:24 on dist-test-slave-p3t5"
I20260812 20:01:24.118813 7442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "2a7d3d01abea41b89bf31e9e46051f99"
format_stamp: "Formatted at 2026-08-12 20:01:24 on dist-test-slave-p3t5"
I20260812 20:01:24.124883 7442 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:01:24.129117 7459 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:24.130882 7442 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:24.131143 7442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "2a7d3d01abea41b89bf31e9e46051f99"
format_stamp: "Formatted at 2026-08-12 20:01:24 on dist-test-slave-p3t5"
I20260812 20:01:24.131404 7442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:24.148645 7442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:24.152890 7442 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:24.153230 7442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:24.175176 7442 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:34851
I20260812 20:01:24.175194 7517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:34851 every 8 connection(s)
I20260812 20:01:24.176927 7442 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:01:24.181728 7518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:24.182476 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7442
I20260812 20:01:24.182690 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:01:24.191110 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564884, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:24.215943 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564884, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:24.230325 7431 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 5
I20260812 20:01:24.234658 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564883, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564884, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:24.245043 7343 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } attempt: 5
Aug 12 20:01:24 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564882, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:24.258286 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: Bootstrap starting.
I20260812 20:01:24.259253 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Bootstrap starting.
I20260812 20:01:24.263080 7518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:24.263286 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:24.264513 7518 log.cc:826] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:24.265496 7431 log.cc:826] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:24.267634 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Bootstrap starting.
I20260812 20:01:24.272132 7343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:24.272156 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: No bootstrap required, opened a new log
I20260812 20:01:24.272158 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: No bootstrap required, opened a new log
I20260812 20:01:24.273744 7343 log.cc:826] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:24.279179 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: No bootstrap required, opened a new log
I20260812 20:01:24.280418 7431 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.280418 7518 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.280791 7431 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:24.280836 7518 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:24.280946 7431 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55baa567a6b24e18aa2aba6888d0f714, State: Initialized, Role: FOLLOWER
I20260812 20:01:24.280997 7518 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7d3d01abea41b89bf31e9e46051f99, State: Initialized, Role: FOLLOWER
I20260812 20:01:24.281746 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.281760 7518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.284946 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.285918 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.286293 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.286448 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.286960 7518 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:24.287009 7343 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.287223 7431 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:24.287367 7343 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:24.287520 7343 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0692a5f31f6042a0ad3df7e96143abd3, State: Initialized, Role: FOLLOWER
I20260812 20:01:24.288261 7343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.292243 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.292555 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.296695 7343 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:24.307611 7295 tablet.cc:2406] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:01:24.328347 7564 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:24.328559 7564 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:24.318675 7442 master_runner.cc:187] Error getting master registration for 127.25.14.62:36151: OK, instance_id { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" instance_seqno: 1786564882820514 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:24.330299 7442 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:24.339919 7576 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 20:01:24.348570 7574 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:24.348744 7574 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:24.355242 7354 master_runner.cc:264] No leader master found from master 55baa567a6b24e18aa2aba6888d0f714
I20260812 20:01:24.355441 7354 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:24.356320 7272 master_runner.cc:264] No leader master found from master 0692a5f31f6042a0ad3df7e96143abd3
I20260812 20:01:24.356489 7272 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:24.401099 7532 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:01:24.401470 7532 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.403656 7532 leader_election.cc:290] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0692a5f31f6042a0ad3df7e96143abd3 (127.25.14.62:36151), 55baa567a6b24e18aa2aba6888d0f714 (127.25.14.61:40801)
I20260812 20:01:24.404728 7316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7d3d01abea41b89bf31e9e46051f99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0692a5f31f6042a0ad3df7e96143abd3" is_pre_election: true
I20260812 20:01:24.404852 7398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7d3d01abea41b89bf31e9e46051f99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55baa567a6b24e18aa2aba6888d0f714" is_pre_election: true
I20260812 20:01:24.405297 7316 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7d3d01abea41b89bf31e9e46051f99 in term 0.
I20260812 20:01:24.405409 7398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7d3d01abea41b89bf31e9e46051f99 in term 0.
W20260812 20:01:24.405896 7465 tablet.cc:2406] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:24.406250 7461 leader_election.cc:304] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0692a5f31f6042a0ad3df7e96143abd3, 2a7d3d01abea41b89bf31e9e46051f99; no voters:
I20260812 20:01:24.407564 7532 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:01:24.407737 7532 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:01:24.407868 7532 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:24.412487 7532 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.414794 7316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7d3d01abea41b89bf31e9e46051f99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0692a5f31f6042a0ad3df7e96143abd3"
I20260812 20:01:24.415055 7316 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:24.415982 7398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a7d3d01abea41b89bf31e9e46051f99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55baa567a6b24e18aa2aba6888d0f714"
I20260812 20:01:24.416208 7398 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:24.418696 7316 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7d3d01abea41b89bf31e9e46051f99 in term 1.
I20260812 20:01:24.419368 7461 leader_election.cc:304] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0692a5f31f6042a0ad3df7e96143abd3, 2a7d3d01abea41b89bf31e9e46051f99; no voters:
I20260812 20:01:24.420635 7398 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7d3d01abea41b89bf31e9e46051f99 in term 1.
I20260812 20:01:24.421329 7532 leader_election.cc:290] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [CANDIDATE]: Term 1 election: Requested vote from peers 0692a5f31f6042a0ad3df7e96143abd3 (127.25.14.62:36151), 55baa567a6b24e18aa2aba6888d0f714 (127.25.14.61:40801)
I20260812 20:01:24.421677 7532 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:01:24.421911 7532 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 1 LEADER]: Becoming Leader. State: Replica: 2a7d3d01abea41b89bf31e9e46051f99, State: Running, Role: LEADER
I20260812 20:01:24.422577 7532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } }
I20260812 20:01:24.428601 7578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7d3d01abea41b89bf31e9e46051f99. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.429102 7578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:24.431288 7580 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:01:24.434954 7580 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:01:24.444416 7398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:24.444505 7316 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:24.445585 7578 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:24.453838 7532 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:24.472003 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7d3d01abea41b89bf31e9e46051f99. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.472319 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.476523 7582 mvcc.cc:204] Tried to move back new op lower bound from 7317769766672154624 to 7317769766600626176. Current Snapshot: MvccSnapshot[applied={T|T < 7317769766672154624}]
I20260812 20:01:24.486017 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7d3d01abea41b89bf31e9e46051f99. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.486337 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.489528 7578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.489856 7578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:24.490361 7578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.490631 7578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:24.491335 7586 mvcc.cc:204] Tried to move back new op lower bound from 7317769766672154624 to 7317769766600626176. Current Snapshot: MvccSnapshot[applied={T|T < 7317769766672154624}]
I20260812 20:01:24.494536 7580 catalog_manager.cc:1383] Generated new cluster ID: 5d7b3e87425342749203e6fbb2171c54
I20260812 20:01:24.494655 7580 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:01:24.498095 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.498349 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.511266 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } }
I20260812 20:01:24.511538 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:24.534765 7580 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:01:24.538338 7580 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:01:24.552846 7580 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Generated new TSK 0
I20260812 20:01:24.554281 7580 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:24.666440 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.103s user 0.000s sys 0.036s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:24.773221 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.107s user 0.001s sys 0.036s
I20260812 20:01:24.774597 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:25.205606 7597 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:25.206053 7597 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:25.206158 7597 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:25.215041 7597 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:25.215246 7597 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:25.226619 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:25.228461 7597 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:25.229971 7597 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:25.241710 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:25.241699 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:25.247067 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:25.247566 7597 server_base.cc:1061] running on GCE node
I20260812 20:01:25.248294 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:25.249579 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:25.250923 7597 hybrid_clock.cc:648] HybridClock initialized: now 1786564885250742 us; error 142 us; skew 500 ppm
I20260812 20:01:25.251302 7597 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:25 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564885, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:25.254952 7597 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:25.256875 7597 webserver.cc:533] Webserver started at http://127.25.14.1:45491/ using document root <none> and password file <none>
I20260812 20:01:25.257759 7597 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:25.257895 7597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:25.258317 7597 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:25.261000 7597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:25.261998 7597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:25.268178 7597 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:01:25.272063 7616 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:25.273663 7597 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:01:25.273888 7597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:25.274117 7597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:25.305591 7597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:25.315608 7597 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:25.315955 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:25.317126 7597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:25.319126 7597 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:25.319244 7597 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:25.319409 7597 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:25.319495 7597 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:25.351172 7574 catalog_manager.cc:1295] Loaded cluster ID: 5d7b3e87425342749203e6fbb2171c54
I20260812 20:01:25.351300 7574 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: loading cluster ID for follower catalog manager: success
I20260812 20:01:25.351513 7576 catalog_manager.cc:1295] Loaded cluster ID: 5d7b3e87425342749203e6fbb2171c54
I20260812 20:01:25.351644 7576 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: loading cluster ID for follower catalog manager: success
I20260812 20:01:25.355484 7574 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: acquiring CA information for follower catalog manager: success
I20260812 20:01:25.357275 7576 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: acquiring CA information for follower catalog manager: success
I20260812 20:01:25.358680 7574 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:25.360545 7576 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:25.369236 7597 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40641
I20260812 20:01:25.369275 7740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40641 every 8 connection(s)
I20260812 20:01:25.370899 7597 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:25.378500 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7597
I20260812 20:01:25.378793 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:01:25 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564885, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:25 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564885, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:25 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564885, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:25.429872 7746 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40801
I20260812 20:01:25.430326 7746 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:25.431545 7746 heartbeater.cc:507] Master 127.25.14.61:40801 requested a full tablet report, sending...
I20260812 20:01:25.435366 7388 ts_manager.cc:194] Registered new tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:25.436308 7743 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34851
I20260812 20:01:25.436506 7743 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:25.437150 7743 heartbeater.cc:507] Master 127.25.14.60:34851 requested a full tablet report, sending...
I20260812 20:01:25.438251 7741 heartbeater.cc:344] Connected to a master server at 127.25.14.62:36151
I20260812 20:01:25.438407 7741 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:25.438947 7741 heartbeater.cc:507] Master 127.25.14.62:36151 requested a full tablet report, sending...
I20260812 20:01:25.439211 7477 ts_manager.cc:194] Registered new tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:25.441246 7306 ts_manager.cc:194] Registered new tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:25.441546 7477 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:37335
I20260812 20:01:25.448108 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:25.625691 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:39223
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:26.023377 7760 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:26.023765 7760 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:26.023862 7760 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:26.033291 7760 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:26.033427 7760 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:26.033486 7760 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:26.033526 7760 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:26.045572 7760 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:26.047609 7760 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:26.049124 7760 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:26.058204 7767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:26.058204 7765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:26.059233 7770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:26.059837 7760 server_base.cc:1061] running on GCE node
I20260812 20:01:26.060753 7760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:26.062588 7760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:26.063882 7760 hybrid_clock.cc:648] HybridClock initialized: now 1786564886063731 us; error 117 us; skew 500 ppm
I20260812 20:01:26.064275 7760 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:26.068033 7760 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:26.070005 7760 webserver.cc:533] Webserver started at http://127.25.14.59:34211/ using document root <none> and password file <none>
I20260812 20:01:26.070942 7760 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:26.071085 7760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:26.071496 7760 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:26.074191 7760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "95c27933148a48648b490d190f62f277"
format_stamp: "Formatted at 2026-08-12 20:01:26 on dist-test-slave-p3t5"
I20260812 20:01:26.075033 7760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "95c27933148a48648b490d190f62f277"
format_stamp: "Formatted at 2026-08-12 20:01:26 on dist-test-slave-p3t5"
I20260812 20:01:26.081125 7760 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260812 20:01:26.085156 7778 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:26.086859 7760 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 20:01:26.087110 7760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "95c27933148a48648b490d190f62f277"
format_stamp: "Formatted at 2026-08-12 20:01:26 on dist-test-slave-p3t5"
I20260812 20:01:26.087339 7760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:26.104315 7760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:26.108733 7760 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:26.109076 7760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:26.131991 7760 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:39223
I20260812 20:01:26.132041 7845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:39223 every 8 connection(s)
I20260812 20:01:26.133893 7760 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:26.138617 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 7760
I20260812 20:01:26.139180 7846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:26.147528 7846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:26.167412 7846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:26.183055 7846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:26.197578 7846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:26.215240 7846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Bootstrap starting.
I20260812 20:01:26.219316 7846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:26.220551 7846 log.cc:826] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:26.223438 7846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: No bootstrap required, opened a new log
I20260812 20:01:26.228801 7846 raft_consensus.cc:359] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } }
I20260812 20:01:26.229188 7846 raft_consensus.cc:385] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:26.229316 7846 raft_consensus.cc:740] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c27933148a48648b490d190f62f277, State: Initialized, Role: FOLLOWER
I20260812 20:01:26.230077 7846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } }
I20260812 20:01:26.231076 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } } }
I20260812 20:01:26.231460 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:26.232203 7846 sys_catalog.cc:565] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:26.243448 7870 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:26.243614 7870 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:26.253705 7760 master_runner.cc:429] Detected that this master 95c27933148a48648b490d190f62f277 is joining an existing cluster
I20260812 20:01:26.253875 7760 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:01:26.254900 7760 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:01:26.255363 7760 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:01:26.256424 7760 tablet_replica.cc:333] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: stopping tablet replica
I20260812 20:01:26.258616 7760 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:39223
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564884, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:26.338918 7477 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:39223: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "95c27933148a48648b490d190f62f277"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 39223
}
attrs {
promote: true
}
}
dest_uuid: "2a7d3d01abea41b89bf31e9e46051f99"
cas_config_opid_index: -1
I20260812 20:01:26.340953 7477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } }
I20260812 20:01:26.345366 7315 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:01:26.345660 7462 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 -> Peer 95c27933148a48648b490d190f62f277 (127.25.14.59:39223): Couldn't send request to peer 95c27933148a48648b490d190f62f277. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:26.345956 7398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:01:26.346155 7532 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:26.346879 7532 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:26.358798 7532 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 95c27933148a48648b490d190f62f277 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.359891 7315 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 95c27933148a48648b490d190f62f277 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.363943 7532 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:39223
I20260812 20:01:26.364962 7760 master.cc:562] Master@127.25.14.59:39223 shutting down...
I20260812 20:01:26.365828 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.366101 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:26.366844 7883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.367238 7883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:26.372906 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.373283 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:01:26.388855 7780 connection.cc:466] server connection from 127.0.0.1:58712 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:58712 (request call id 14) could send its response
I20260812 20:01:26.392284 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.392550 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:26.392870 7398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 95c27933148a48648b490d190f62f277 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.397301 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.397778 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:26.404222 7760 master.cc:584] Master@127.25.14.59:39223 shutdown complete.
I20260812 20:01:26.404341 7760 master_runner.cc:306] Clearing existing system tablet
I20260812 20:01:26.404759 7760 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:26.404902 7760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:26.408883 7760 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260812 20:01:26.411052 7894 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:26.414503 7760 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:26.414847 7760 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260812 20:01:26.414954 7760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "95c27933148a48648b490d190f62f277"
format_stamp: "Formatted at 2026-08-12 20:01:26 on dist-test-slave-p3t5"
I20260812 20:01:26.419579 7760 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:01:26.427404 7760 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:01:26.427690 7760 log.cc:1199] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:01:26.428301 7760 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Deleting consensus metadata
I20260812 20:01:26.428656 7760 master_runner.cc:316] Copying system tablet from 127.25.14.60:34851
I20260812 20:01:26.431504 7760 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:34851
I20260812 20:01:26.445403 7743 heartbeater.cc:499] Master 127.25.14.60:34851 was elected leader, sending a full tablet report...
I20260812 20:01:26.450040 7500 tablet_copy_service.cc:140] P 2a7d3d01abea41b89bf31e9e46051f99: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 95c27933148a48648b490d190f62f277 ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:52188)
I20260812 20:01:26.450414 7500 tablet_copy_service.cc:161] P 2a7d3d01abea41b89bf31e9e46051f99: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 95c27933148a48648b490d190f62f277 at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:52188: session id = 95c27933148a48648b490d190f62f277-00000000000000000000000000000000
I20260812 20:01:26.455295 7500 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:01:26.458811 7760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:01:26.466517 7760 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: tablet copy: Starting download of 0 data blocks...
I20260812 20:01:26.466897 7760 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: tablet copy: Starting download of 1 WAL segments...
I20260812 20:01:26.470008 7760 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:01:26.475021 7500 tablet_copy_service.cc:342] P 2a7d3d01abea41b89bf31e9e46051f99: Request end of tablet copy session 95c27933148a48648b490d190f62f277-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:52188
I20260812 20:01:26.475282 7500 tablet_copy_service.cc:434] P 2a7d3d01abea41b89bf31e9e46051f99: ending tablet copy session 95c27933148a48648b490d190f62f277-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 95c27933148a48648b490d190f62f277
W20260812 20:01:26.482805 7760 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:01:26.487135 7760 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:26.490208 7902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:26.490293 7901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:26.491559 7905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:26.494519 7760 server_base.cc:1061] running on GCE node
I20260812 20:01:26.494855 7760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:26.495642 7760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:26.496851 7760 hybrid_clock.cc:648] HybridClock initialized: now 1786564886496716 us; error 109 us; skew 500 ppm
I20260812 20:01:26.497136 7760 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564886, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:26.499246 7760 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:26.500525 7760 webserver.cc:533] Webserver started at http://127.25.14.59:40563/ using document root <none> and password file <none>
I20260812 20:01:26.500852 7760 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:26.500957 7760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:26.503474 7760 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260812 20:01:26.505609 7913 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:26.506152 7760 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 20:01:26.506304 7760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "95c27933148a48648b490d190f62f277"
format_stamp: "Formatted at 2026-08-12 20:01:26 on dist-test-slave-p3t5"
I20260812 20:01:26.506515 7760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:26.527781 7760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:26.531158 7760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:26.551544 7760 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:39223
I20260812 20:01:26.551580 7979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:39223 every 8 connection(s)
I20260812 20:01:26.552601 7760 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:26.556349 7980 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:26.560189 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Bootstrap starting.
I20260812 20:01:26.576844 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:26.577382 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: Bootstrap complete.
I20260812 20:01:26.578444 7980 raft_consensus.cc:359] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } }
I20260812 20:01:26.578707 7980 raft_consensus.cc:740] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95c27933148a48648b490d190f62f277, State: Initialized, Role: LEARNER
I20260812 20:01:26.578967 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } }
I20260812 20:01:26.580056 7980 sys_catalog.cc:565] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:26.579893 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.580289 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:26.589331 8006 catalog_manager.cc:1295] Loaded cluster ID: 5d7b3e87425342749203e6fbb2171c54
I20260812 20:01:26.589470 8006 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: loading cluster ID for follower catalog manager: success
I20260812 20:01:26.592332 8006 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: acquiring CA information for follower catalog manager: success
I20260812 20:01:26.594523 8006 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:26.612785 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7597
Aug 12 20:01:26 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:01:26.631896 7736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:26.724032 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7597
I20260812 20:01:26.749581 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--webserver_port=45491
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:26.836752 7948 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:01:26.837536 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7d3d01abea41b89bf31e9e46051f99. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:26.837839 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:01:27.146301 8010 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:27.146741 8010 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:27.146926 8010 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:27.155666 8010 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:27.155978 8010 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:27.167733 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45491
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:27.169546 8010 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:27.171110 8010 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:27.182180 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:27.183300 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:27.183703 8010 server_base.cc:1061] running on GCE node
W20260812 20:01:27.183795 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:27.184590 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:27.185945 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:27.189327 8010 hybrid_clock.cc:648] HybridClock initialized: now 1786564887189103 us; error 177 us; skew 500 ppm
I20260812 20:01:27.189778 8010 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:27.193462 8010 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:27.195513 8010 webserver.cc:533] Webserver started at http://127.25.14.1:45491/ using document root <none> and password file <none>
I20260812 20:01:27.196364 8010 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:27.196538 8010 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:27.202975 8010 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:01:27.206210 8025 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:27.207883 8010 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 20:01:27.208142 8010 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:27.208992 8010 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:27.222764 7883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: true } } }
I20260812 20:01:27.223073 7883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:27.223728 8010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:27.224524 7883 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99: attempting to promote NON_VOTER 95c27933148a48648b490d190f62f277 to VOTER
I20260812 20:01:27.225353 7883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } }
I20260812 20:01:27.227857 7948 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 1 LEARNER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:27.227901 7315 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:27.228051 7398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Refusing update from remote peer 2a7d3d01abea41b89bf31e9e46051f99: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:27.228675 8032 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:27.229035 7883 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:27.229295 8030 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:27.229496 8010 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:27.229890 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:27.232139 8010 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:27.234082 7398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 95c27933148a48648b490d190f62f277 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.234651 8010 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:27.234763 8010 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:27.234867 8010 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:27.234954 8010 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:27.234647 7948 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 95c27933148a48648b490d190f62f277 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.233220 8032 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 95c27933148a48648b490d190f62f277 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.240799 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.241066 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c27933148a48648b490d190f62f277 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:27.240926 8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.241184 8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55baa567a6b24e18aa2aba6888d0f714 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:27.243341 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.243579 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:27.243561 8030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.243836 8030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:27.244005 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.244230 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:27.244345 7883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.244527 7883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:27.246378 7315 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 95c27933148a48648b490d190f62f277 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.251293 8057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:27.251621 8057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:27.282454 8010 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40641
I20260812 20:01:27.282552 8166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40641 every 8 connection(s)
I20260812 20:01:27.284356 8010 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:27.291719 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8010
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:27.344025 8170 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40801
I20260812 20:01:27.344293 8168 heartbeater.cc:344] Connected to a master server at 127.25.14.59:39223
I20260812 20:01:27.344542 8168 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:27.344604 8167 heartbeater.cc:344] Connected to a master server at 127.25.14.62:36151
I20260812 20:01:27.344740 8167 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:27.344538 8170 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:27.346060 8167 heartbeater.cc:507] Master 127.25.14.62:36151 requested a full tablet report, sending...
I20260812 20:01:27.346086 8169 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34851
I20260812 20:01:27.346280 8169 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:27.346403 8170 heartbeater.cc:507] Master 127.25.14.61:40801 requested a full tablet report, sending...
I20260812 20:01:27.346848 8169 heartbeater.cc:507] Master 127.25.14.60:34851 requested a full tablet report, sending...
I20260812 20:01:27.347365 8168 heartbeater.cc:507] Master 127.25.14.59:39223 requested a full tablet report, sending...
I20260812 20:01:27.348656 7388 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:27.348680 7477 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:27.349210 7934 ts_manager.cc:194] Registered new tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:27.349895 7477 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:59931
I20260812 20:01:27.350867 7305 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:27.351652 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:27.360513 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:27.360721 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7272
I20260812 20:01:27.361770 7338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:27.479244 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7272
I20260812 20:01:27.509650 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36151
--webserver_interface=127.25.14.62
--webserver_port=40453
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:27.735492 7461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:36151: connect: Connection refused (error 111)
W20260812 20:01:27.738446 7461 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 -> Peer 0692a5f31f6042a0ad3df7e96143abd3 (127.25.14.62:36151): Couldn't send request to peer 0692a5f31f6042a0ad3df7e96143abd3. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:36151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:01:27.913026 8180 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:27.913528 8180 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:27.913705 8180 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:27.922700 8180 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:27.922827 8180 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:27.922884 8180 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:27.922933 8180 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:27.934538 8180 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=40453
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:27.936321 8180 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:27.937971 8180 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:27.947431 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:27.947604 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:27.947897 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:27.949079 8180 server_base.cc:1061] running on GCE node
I20260812 20:01:27.950055 8180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:27.952006 8180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:27.953334 8180 hybrid_clock.cc:648] HybridClock initialized: now 1786564887953137 us; error 159 us; skew 500 ppm
I20260812 20:01:27.953788 8180 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:27 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564887, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:27.957307 8180 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:27.959273 8180 webserver.cc:533] Webserver started at http://127.25.14.62:40453/ using document root <none> and password file <none>
I20260812 20:01:27.960095 8180 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:27.960264 8180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:27.966634 8180 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260812 20:01:27.970181 8200 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:27.971908 8180 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 20:01:27.972177 8180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "0692a5f31f6042a0ad3df7e96143abd3"
format_stamp: "Formatted at 2026-08-12 20:01:22 on dist-test-slave-p3t5"
I20260812 20:01:27.972999 8180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:28.013103 8180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:28.017727 8180 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:28.018010 8180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:28.040423 8180 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:36151
I20260812 20:01:28.040450 8265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:36151 every 8 connection(s)
I20260812 20:01:28.042341 8180 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:28.043948 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8180
I20260812 20:01:28.044281 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8010
I20260812 20:01:28.051504 8266 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:28.055856 8266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Bootstrap starting.
I20260812 20:01:28.062538 8162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:28.076067 8266 log.cc:826] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:28.087477 8266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:28.088253 8266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: Bootstrap complete.
I20260812 20:01:28.095917 8266 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } }
I20260812 20:01:28.096753 8266 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0692a5f31f6042a0ad3df7e96143abd3, State: Initialized, Role: FOLLOWER
I20260812 20:01:28.097566 8266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } }
I20260812 20:01:28.098663 8274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:28.099045 8274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:28.100014 8266 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:28.124359 8286 catalog_manager.cc:1295] Loaded cluster ID: 5d7b3e87425342749203e6fbb2171c54
I20260812 20:01:28.124512 8286 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: loading cluster ID for follower catalog manager: success
I20260812 20:01:28.129678 8286 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: acquiring CA information for follower catalog manager: success
I20260812 20:01:28.132831 8286 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:28.190407 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8010
I20260812 20:01:28.194844 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:28.195348 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:28.202517 8274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7d3d01abea41b89bf31e9e46051f99. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:28.202780 8274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0692a5f31f6042a0ad3df7e96143abd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:28.220113 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--webserver_port=45491
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:28.578637 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a7d3d01abea41b89bf31e9e46051f99" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0692a5f31f6042a0ad3df7e96143abd3" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36151 } } peers { permanent_uuid: "55baa567a6b24e18aa2aba6888d0f714" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 40801 } } peers { permanent_uuid: "2a7d3d01abea41b89bf31e9e46051f99" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 34851 } } peers { permanent_uuid: "95c27933148a48648b490d190f62f277" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 39223 } attrs { promote: false } } }
I20260812 20:01:28.578974 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 [sys.catalog]: This master's current role is: LEADER
W20260812 20:01:28.623983 8303 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:28.624441 8303 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:28.624614 8303 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:28.633589 8303 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:28.633932 8303 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:28.645274 8303 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45491
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:28.647280 8303 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:28.649001 8303 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:28.660125 8309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:28.660261 8311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:28.660151 8308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:28.661676 8303 server_base.cc:1061] running on GCE node
I20260812 20:01:28.662460 8303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:28.663800 8303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:28.665184 8303 hybrid_clock.cc:648] HybridClock initialized: now 1786564888665007 us; error 134 us; skew 500 ppm
I20260812 20:01:28.665506 8303 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:28 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564888, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:28.669488 8303 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:28.671346 8303 webserver.cc:533] Webserver started at http://127.25.14.1:45491/ using document root <none> and password file <none>
I20260812 20:01:28.672128 8303 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:28.672237 8303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:28.678455 8303 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:01:28.681747 8319 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:28.683534 8303 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:01:28.683760 8303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:28.684612 8303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:28.717233 8303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:28.721346 8303 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:28.721725 8303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:28.722939 8303 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:28.724754 8303 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:28.724879 8303 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:28.725014 8303 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:28.725121 8303 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:28.762609 8303 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40641
I20260812 20:01:28.762678 8455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40641 every 8 connection(s)
I20260812 20:01:28.764358 8303 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:28.767434 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8303
Aug 12 20:01:28 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564888, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:28 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564888, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:28 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564888, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:28 dist-test-slave-p3t5 krb5kdc[7242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564888, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:28.820837 8456 heartbeater.cc:344] Connected to a master server at 127.25.14.62:36151
I20260812 20:01:28.821300 8456 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:28.821564 8459 heartbeater.cc:344] Connected to a master server at 127.25.14.61:40801
I20260812 20:01:28.821729 8459 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:28.822635 8456 heartbeater.cc:507] Master 127.25.14.62:36151 requested a full tablet report, sending...
I20260812 20:01:28.823889 8459 heartbeater.cc:507] Master 127.25.14.61:40801 requested a full tablet report, sending...
I20260812 20:01:28.825196 7388 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:28.825575 8222 ts_manager.cc:194] Registered new tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:28.827946 8457 heartbeater.cc:344] Connected to a master server at 127.25.14.59:39223
I20260812 20:01:28.828125 8457 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:28.828682 8457 heartbeater.cc:507] Master 127.25.14.59:39223 requested a full tablet report, sending...
I20260812 20:01:28.830299 7933 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:28.835279 8458 heartbeater.cc:344] Connected to a master server at 127.25.14.60:34851
I20260812 20:01:28.835481 8458 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:28.835943 8458 heartbeater.cc:507] Master 127.25.14.60:34851 requested a full tablet report, sending...
I20260812 20:01:28.837431 7477 ts_manager.cc:194] Re-registered known tserver with Master: cc230f410cbc4da8bb134a4b097bcdd2 (127.25.14.1:40641)
I20260812 20:01:28.838703 7477 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:56219
W20260812 20:01:28.879520 25656 mini_kdc.cc:336] Time spent kdestroy: real 0.112s user 0.002s sys 0.037s
I20260812 20:01:28.883419 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8303
W20260812 20:01:28.885097 8472 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:28.880756 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.25.14.1:40641)
0812 20:01:28.881865 (+ 1109us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:28.881873 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:28.881909 (+ 36us) server_negotiation.cc:408] Connection header received
0812 20:01:28.881980 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:28.881989 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:28.882224 (+ 235us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:28.882412 (+ 188us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:28.883374 (+ 962us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49766: BlockingRecv error: recv got EOF from 127.0.0.1:49766 (error 108)
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":706,"threads_started":1}
W20260812 20:01:28.886037 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8303: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40641: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:28.886225 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8303
I20260812 20:01:28.912884 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--webserver_port=45491
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:29.308221 8474 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:29.308727 8474 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:29.308902 8474 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:01:29.308969 8474 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:29.318192 8474 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:29.318482 8474 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:29.330940 8474 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45491
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:29.333151 8474 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:29.335014 8474 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:29.346886 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:29.346947 8484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:29.347330 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:29.348340 8474 server_base.cc:1061] running on GCE node
I20260812 20:01:29.348958 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:29.350242 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:29.351657 8474 hybrid_clock.cc:648] HybridClock initialized: now 1786564889351479 us; error 122 us; skew 500 ppm
I20260812 20:01:29.352018 8474 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:29 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564889, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:29.355624 8474 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:29.357992 8474 webserver.cc:533] Webserver started at http://127.25.14.1:45491/ using document root <none> and password file <none>
I20260812 20:01:29.358822 8474 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:29.358968 8474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:29.366050 8474 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260812 20:01:29.369365 8493 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:29.370955 8474 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:01:29.371160 8474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:29.371930 8474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:29.384399 8474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:29.388753 8474 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:29.389076 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:29.390142 8474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:29.391777 8474 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:29.391886 8474 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:29.392045 8474 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:29.392118 8474 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:01:29.392513 8474 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:01:29.392603 8474 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:01:29.392711 8474 fault_injection.cc:59] injecting fault for kudu (pid 8474): FLAGS_fault_before_start (process will exit)
I20260812 20:01:29.416783 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:40641
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851
--webserver_port=45491
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:29.803896 8503 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:29.804338 8503 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:29.804519 8503 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:29.813889 8503 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:29.814167 8503 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:29.825453 8503 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36385,127.25.14.21:36001,127.25.14.22:45873,127.25.14.23:35031,127.25.14.24:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45491
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36151,127.25.14.61:40801,127.25.14.60:34851,127.25.14.59:39223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:29.827334 8503 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:29.828909 8503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:29.840102 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:29.840577 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:29.840129 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:29.841317 8503 server_base.cc:1061] running on GCE node
I20260812 20:01:29.842046 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:01:29.843339 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:01:29.844724 8503 hybrid_clock.cc:648] HybridClock initialized: now 1786564889844580 us; error 105 us; skew 500 ppm
I20260812 20:01:29.845111 8503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:29 dist-test-slave-p3t5 krb5kdc[7242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564889, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:29.848939 8503 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:29.850859 8503 webserver.cc:533] Webserver started at http://127.25.14.1:45491/ using document root <none> and password file <none>
I20260812 20:01:29.851778 8503 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:29.851891 8503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:29.858163 8503 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:29.861496 8520 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:29.863205 8503 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:01:29.863412 8503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "cc230f410cbc4da8bb134a4b097bcdd2"
format_stamp: "Formatted at 2026-08-12 20:01:25 on dist-test-slave-p3t5"
I20260812 20:01:29.864238 8503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:29.878134 8503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:29.882661 8503 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:29.882969 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:29.884138 8503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:29.885957 8503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:29.886086 8503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:29.886219 8503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:29.886309 8503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:29.923892 8503 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:40641
I20260812 20:01:29.923990 8654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:40641 every 8 connection(s)
I20260812 20:01:29.925598 8503 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:29.928661 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 8503
I20260812 20:01:29.929031 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8503
W20260812 20:01:29.982764 8671 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:29.971207 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.25.14.1:40641)
0812 20:01:29.977983 (+ 6776us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:29.977993 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:29.978027 (+ 34us) server_negotiation.cc:408] Connection header received
0812 20:01:29.978080 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:29.978088 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:29.978762 (+ 674us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:29.979022 (+ 260us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:29.981092 (+ 2070us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49782: BlockingRecv error: recv got EOF from 127.0.0.1:49782 (error 108)
Metrics: {"server-negotiator.queue_time_us":6564,"thread_start_us":7187,"threads_started":1}
W20260812 20:01:29.983686 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8503: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:40641: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:29.983855 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8503
W20260812 20:01:30.011722 8664 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:29.931095 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:52997 (local address 127.25.14.62:36151)
0812 20:01:29.968136 (+ 37041us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:29.968146 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:29.971509 (+ 3363us) server_negotiation.cc:408] Connection header received
0812 20:01:29.971610 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:29.971620 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:29.971730 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:29.971866 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:29.978137 (+ 6271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:29.979686 (+ 1549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:29.982203 (+ 2517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:29.982697 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:30.010784 (+ 28087us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:52997: BlockingRecv error: recv got EOF from 127.25.14.1:52997 (error 108)
Metrics: {"server-negotiator.queue_time_us":36843,"thread_start_us":37073,"threads_started":1}
W20260812 20:01:30.011722 8660 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:29.929291 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:53475 (local address 127.25.14.60:34851)
0812 20:01:29.989809 (+ 60518us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:29.989820 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:29.989855 (+ 35us) server_negotiation.cc:408] Connection header received
0812 20:01:29.989961 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:29.989972 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:29.990088 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:29.990248 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:30.010973 (+ 20725us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:53475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":60276,"thread_start_us":60143,"threads_started":1}
W20260812 20:01:30.011735 8665 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:29.931538 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:36799 (local address 127.25.14.61:40801)
0812 20:01:29.965399 (+ 33861us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:29.965410 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:29.970097 (+ 4687us) server_negotiation.cc:408] Connection header received
0812 20:01:29.970204 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:29.970213 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:29.970352 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:29.970495 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:30.010695 (+ 40200us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:36799: BlockingRecv error: recv got EOF from 127.25.14.1:36799 (error 108)
Metrics: {"server-negotiator.queue_time_us":33647,"thread_start_us":33895,"threads_started":1}
W20260812 20:01:30.011746 8669 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:29.936384 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:56971 (local address 127.25.14.59:39223)
0812 20:01:29.936980 (+ 596us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:29.936989 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:30.010660 (+ 73671us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:56971: BlockingRecv error: recv got EOF from 127.25.14.1:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":286,"threads_started":1}
I20260812 20:01:30.012205 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8180
W20260812 20:01:30.014784 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 8180: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:36151: server requires authentication, but client does not have Kerberos credentials available
W20260812 20:01:30.014891 8664 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:30.013108 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.25.14.62:36151)
0812 20:01:30.013307 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:30.013314 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:30.013426 (+ 112us) server_negotiation.cc:408] Connection header received
0812 20:01:30.013690 (+ 264us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:30.013698 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:30.013800 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:30.013965 (+ 165us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:30.014691 (+ 726us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54428: BlockingRecv error: recv got EOF from 127.0.0.1:54428 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 20:01:30.014945 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 8180
I20260812 20:01:30.045827 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7354
I20260812 20:01:30.047220 7426 generic_service.cc:196] Checking for leaks (request via RPC)
W20260812 20:01:30.143299 7461 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2a7d3d01abea41b89bf31e9e46051f99 -> Peer 0692a5f31f6042a0ad3df7e96143abd3 (127.25.14.62:36151): Couldn't send request to peer 0692a5f31f6042a0ad3df7e96143abd3. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:36151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:30.164320 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7354
I20260812 20:01:30.195042 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7442
I20260812 20:01:30.196476 7512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:30.318312 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7442
I20260812 20:01:30.348409 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 7760
I20260812 20:01:30.349864 7974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:30.468778 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 7760
2026-08-12T20:01:30Z chronyd exiting
2026-08-12T20:01:30Z chronyd exiting
2026-08-12T20:01:30Z chronyd exiting
2026-08-12T20:01:30Z chronyd exiting
2026-08-12T20:01:30Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/8 (11192 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/9
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:01:30 dist-test-slave-p3t5 krb5kdc[8679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:01:30 dist-test-slave-p3t5 krb5kdc[8679](info): set up 2 sockets
Aug 12 20:01:30 dist-test-slave-p3t5 krb5kdc[8679](info): commencing operation
krb5kdc: starting...
W20260812 20:01:32.772866 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.202s user 0.002s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:01:33 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 20:01:40: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 20:01:43 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564903, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:01:49 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564909, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:01:49 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564909, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:50.021458 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:42637
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:50.416280 9040 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:50.416669 9040 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:50.416838 9040 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:50.425719 9040 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:50.425844 9040 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:50.425971 9040 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:50.426057 9040 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:50.437335 9040 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:50.439271 9040 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:50.440801 9040 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:50.450110 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:50.450093 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:50.451053 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:50.452648 9040 server_base.cc:1061] running on GCE node
I20260812 20:01:50.453656 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:50.453879 9040 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:50.453990 9040 hybrid_clock.cc:648] HybridClock initialized: now 1786564910453964 us; error 0 us; skew 500 ppm
I20260812 20:01:50.454397 9040 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:50 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:50.458146 9040 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:50.460395 9040 webserver.cc:533] Webserver started at http://127.25.14.62:37979/ using document root <none> and password file <none>
I20260812 20:01:50.461191 9040 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:50.461295 9040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:50.461695 9040 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:50.464279 9040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "524da588370e4a3bbc1da01a524832c4"
format_stamp: "Formatted at 2026-08-12 20:01:50 on dist-test-slave-p3t5"
I20260812 20:01:50.465067 9040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "524da588370e4a3bbc1da01a524832c4"
format_stamp: "Formatted at 2026-08-12 20:01:50 on dist-test-slave-p3t5"
I20260812 20:01:50.471132 9040 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20260812 20:01:50.474944 9060 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:50.476470 9040 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 20:01:50.476686 9040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "524da588370e4a3bbc1da01a524832c4"
format_stamp: "Formatted at 2026-08-12 20:01:50 on dist-test-slave-p3t5"
I20260812 20:01:50.476914 9040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:50.503197 9040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:50.508615 9040 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:50.508961 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:50.532150 9040 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:42637
I20260812 20:01:50.532178 9119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:42637 every 8 connection(s)
I20260812 20:01:50.533927 9040 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:50.535674 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9040
I20260812 20:01:50.535889 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:01:50.539759 9120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:50.549324 9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:50 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:50.587634 9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:50.590000 9063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
W20260812 20:01:50.592100 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:36639: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:50.641577 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } attempt: 1
W20260812 20:01:50.644055 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:36639: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:01:50.712355 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } attempt: 2
W20260812 20:01:50.714869 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:36639: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:50.717082 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:36639
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:50.806176 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } attempt: 3
W20260812 20:01:50.808576 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:36639: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
I20260812 20:01:50.951915 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } attempt: 4
W20260812 20:01:50.954594 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:36639: Network error: Client connection negotiation failed: client connection to 127.25.14.61:36639: connect: Connection refused (error 111)
W20260812 20:01:51.103885 9130 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:51.104364 9130 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:51.104557 9130 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:51.113401 9130 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:51.113528 9130 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:51.113627 9130 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:51.113742 9130 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:51.124923 9130 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:51.126798 9130 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:51.128242 9130 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:51.137117 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:51.138088 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:51.138199 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:51.138743 9130 server_base.cc:1061] running on GCE node
I20260812 20:01:51.139727 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:51.139937 9130 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:51.140053 9130 hybrid_clock.cc:648] HybridClock initialized: now 1786564911140008 us; error 0 us; skew 500 ppm
I20260812 20:01:51.140451 9130 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:51.144165 9130 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:01:51.146327 9130 webserver.cc:533] Webserver started at http://127.25.14.61:46235/ using document root <none> and password file <none>
I20260812 20:01:51.147131 9130 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:51.147231 9130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:51.147541 9130 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:51.150694 9130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "def8d6607a144030aa1972bf4a1b39ff"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.151521 9130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "def8d6607a144030aa1972bf4a1b39ff"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.157786 9130 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 20:01:51.161863 9145 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:51.163482 9130 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 20:01:51.163718 9130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "def8d6607a144030aa1972bf4a1b39ff"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.163980 9130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:51.192194 9130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:51.196605 9130 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:51.196943 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:51.220053 9130 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:36639
I20260812 20:01:51.220086 9205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:36639 every 8 connection(s)
I20260812 20:01:51.221959 9130 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:01:51.226797 9206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:51.229959 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9130
I20260812 20:01:51.230180 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:01:51.235761 9206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:51.253876 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } attempt: 5
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:51.264846 9206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
I20260812 20:01:51.277470 9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:51.280093 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
I20260812 20:01:51.283965 9206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:01:51.285267 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
W20260812 20:01:51.287521 9206 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:51.331436 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 1
W20260812 20:01:51.333742 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
I20260812 20:01:51.336899 9206 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 1
W20260812 20:01:51.339166 9206 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:01:51.402042 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 2
W20260812 20:01:51.404517 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
I20260812 20:01:51.407416 9206 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 2
W20260812 20:01:51.409420 9206 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:51.418102 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43461
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:51.500756 9206 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 3
W20260812 20:01:51.503197 9206 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
I20260812 20:01:51.510798 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 3
W20260812 20:01:51.513052 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
I20260812 20:01:51.646521 9206 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 4
W20260812 20:01:51.648933 9206 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
I20260812 20:01:51.690349 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 4
W20260812 20:01:51.692864 9120 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:43461: Network error: Client connection negotiation failed: client connection to 127.25.14.60:43461: connect: Connection refused (error 111)
W20260812 20:01:51.814632 9217 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:51.815038 9217 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:51.815204 9217 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:51.823882 9217 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:51.824007 9217 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:51.824113 9217 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:51.824206 9217 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:51.835690 9217 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:51.837569 9217 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:51.839267 9217 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:51.848934 9223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:51.849340 9222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:51.850190 9217 server_base.cc:1061] running on GCE node
W20260812 20:01:51.851133 9225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:51.852150 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:51.852377 9217 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:51.852494 9217 hybrid_clock.cc:648] HybridClock initialized: now 1786564911852468 us; error 0 us; skew 500 ppm
I20260812 20:01:51.852912 9217 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:51.856834 9217 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:01:51.859198 9217 webserver.cc:533] Webserver started at http://127.25.14.60:41331/ using document root <none> and password file <none>
I20260812 20:01:51.859995 9217 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:51.860102 9217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:51.860436 9217 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:51.863219 9217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "481bdb5bb5344312be9764d8ff127d5a"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.864064 9217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "481bdb5bb5344312be9764d8ff127d5a"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.870507 9217 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260812 20:01:51.874702 9232 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:51.876412 9217 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:01:51.876624 9217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "481bdb5bb5344312be9764d8ff127d5a"
format_stamp: "Formatted at 2026-08-12 20:01:51 on dist-test-slave-p3t5"
I20260812 20:01:51.876808 9217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:51.894529 9217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:51.903427 9217 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:51.903714 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:51.927172 9217 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:43461
I20260812 20:01:51.927193 9290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:43461 every 8 connection(s)
I20260812 20:01:51.928956 9217 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:01:51.930711 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9217
I20260812 20:01:51.930919 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:01:51.935490 9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:51.943881 9291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:51.948232 9206 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 5
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:51.970168 9120 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } attempt: 5
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564910, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:51.982795 9291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:51.986871 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Bootstrap starting.
I20260812 20:01:51.993214 9206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:51.995929 9206 log.cc:826] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:52.000883 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Bootstrap starting.
I20260812 20:01:52.001768 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: No bootstrap required, opened a new log
Aug 12 20:01:51 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:52.005806 9120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:52.007448 9120 log.cc:826] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:52.013522 9206 raft_consensus.cc:359] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.014201 9206 raft_consensus.cc:385] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:52.014452 9206 raft_consensus.cc:740] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def8d6607a144030aa1972bf4a1b39ff, State: Initialized, Role: FOLLOWER
I20260812 20:01:52.015693 9206 consensus_queue.cc:260] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
Aug 12 20:01:52 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:52.018922 9206 sys_catalog.cc:565] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:52.019312 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.019721 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.026396 9291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:01:52.029452 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: No bootstrap required, opened a new log
I20260812 20:01:52.037711 9120 raft_consensus.cc:359] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.038117 9120 raft_consensus.cc:385] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:52.038259 9120 raft_consensus.cc:740] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524da588370e4a3bbc1da01a524832c4, State: Initialized, Role: FOLLOWER
Aug 12 20:01:52 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:52.039950 9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.041587 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.042060 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.042773 9120 sys_catalog.cc:565] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:52.055116 9291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: Bootstrap starting.
I20260812 20:01:52.051590 9130 master_runner.cc:187] Error getting master registration for 127.25.14.62:42637: OK, instance_id { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" instance_seqno: 1786564910525974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:52.058892 9130 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:52.060375 9291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:52.061792 9291 log.cc:826] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: Log is configured to *not* fsync() on all Append() calls
W20260812 20:01:52.065479 9322 catalog_manager.cc:1594] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:52.065737 9322 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:52.071902 9291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: No bootstrap required, opened a new log
I20260812 20:01:52.083658 9291 raft_consensus.cc:359] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.084230 9291 raft_consensus.cc:385] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:52.084424 9291 raft_consensus.cc:740] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481bdb5bb5344312be9764d8ff127d5a, State: Initialized, Role: FOLLOWER
I20260812 20:01:52.085362 9291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.091361 9339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
W20260812 20:01:52.093027 9337 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:52.093223 9337 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:52.086782 9040 master_runner.cc:187] Error getting master registration for 127.25.14.60:43461: OK, instance_id { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" instance_seqno: 1786564911920570 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:01:52.094861 9040 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:01:52.096109 9291 sys_catalog.cc:565] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:52.097800 9339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:01:52.127549 9351 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 20:01:52.132121 9217 master_runner.cc:264] No leader master found from master 481bdb5bb5344312be9764d8ff127d5a
I20260812 20:01:52.132316 9217 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 20:01:52.127849 9351 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 20:01:52.156153 9239 tablet.cc:2406] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:01:52.200187 9153 tablet.cc:2406] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:52.229981 9308 raft_consensus.cc:493] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:01:52.230332 9308 raft_consensus.cc:515] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.232559 9308 leader_election.cc:290] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524da588370e4a3bbc1da01a524832c4 (127.25.14.62:42637), 481bdb5bb5344312be9764d8ff127d5a (127.25.14.60:43461)
I20260812 20:01:52.233557 9263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "def8d6607a144030aa1972bf4a1b39ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481bdb5bb5344312be9764d8ff127d5a" is_pre_election: true
I20260812 20:01:52.234169 9263 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def8d6607a144030aa1972bf4a1b39ff in term 0.
I20260812 20:01:52.235076 9148 leader_election.cc:304] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481bdb5bb5344312be9764d8ff127d5a, def8d6607a144030aa1972bf4a1b39ff; no voters:
I20260812 20:01:52.236260 9094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "def8d6607a144030aa1972bf4a1b39ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524da588370e4a3bbc1da01a524832c4" is_pre_election: true
I20260812 20:01:52.236812 9094 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def8d6607a144030aa1972bf4a1b39ff in term 0.
I20260812 20:01:52.237936 9308 raft_consensus.cc:2804] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:01:52.238116 9308 raft_consensus.cc:493] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:01:52.238224 9308 raft_consensus.cc:3060] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:52.242776 9308 raft_consensus.cc:515] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.247332 9094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "def8d6607a144030aa1972bf4a1b39ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524da588370e4a3bbc1da01a524832c4"
I20260812 20:01:52.247586 9094 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:01:52.252122 9094 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def8d6607a144030aa1972bf4a1b39ff in term 1.
I20260812 20:01:52.252939 9149 leader_election.cc:304] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524da588370e4a3bbc1da01a524832c4, def8d6607a144030aa1972bf4a1b39ff; no voters:
I20260812 20:01:52.258206 9263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "def8d6607a144030aa1972bf4a1b39ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481bdb5bb5344312be9764d8ff127d5a"
I20260812 20:01:52.258512 9263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:01:52.262768 9066 tablet.cc:2406] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:01:52.267112 9263 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def8d6607a144030aa1972bf4a1b39ff in term 1.
I20260812 20:01:52.272385 9308 leader_election.cc:290] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [CANDIDATE]: Term 1 election: Requested vote from peers 524da588370e4a3bbc1da01a524832c4 (127.25.14.62:42637), 481bdb5bb5344312be9764d8ff127d5a (127.25.14.60:43461)
I20260812 20:01:52.273029 9308 raft_consensus.cc:2804] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:01:52.274067 9308 raft_consensus.cc:697] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 1 LEADER]: Becoming Leader. State: Replica: def8d6607a144030aa1972bf4a1b39ff, State: Running, Role: LEADER
I20260812 20:01:52.274986 9308 consensus_queue.cc:237] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } }
I20260812 20:01:52.297081 9353 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader def8d6607a144030aa1972bf4a1b39ff. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.297713 9353 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:52.304911 9355 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:01:52.307817 9355 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:01:52.327158 9263 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:52.328593 9353 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:52.336380 9094 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:01:52.340974 9353 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:01:52.365127 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def8d6607a144030aa1972bf4a1b39ff. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.365446 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.379139 9356 mvcc.cc:204] Tried to move back new op lower bound from 7317769880877006848 to 7317769880719355904. Current Snapshot: MvccSnapshot[applied={T|T < 7317769880877006848}]
I20260812 20:01:52.395313 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.395692 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:52.401074 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.401372 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.405507 9355 catalog_manager.cc:1383] Generated new cluster ID: 0833137b24d8420fb4f4a2a536e57642
I20260812 20:01:52.405669 9355 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:01:52.422912 9339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader def8d6607a144030aa1972bf4a1b39ff. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.423199 9339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.435333 9353 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.436152 9353 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:52.438114 9357 mvcc.cc:204] Tried to move back new op lower bound from 7317769880877006848 to 7317769880719355904. Current Snapshot: MvccSnapshot[applied={T|T < 7317769880877006848}]
I20260812 20:01:52.446171 9339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } }
I20260812 20:01:52.446483 9339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:52.478003 9355 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:01:52.483747 9355 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:01:52.509855 9355 catalog_manager.cc:6092] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Generated new TSK 0
I20260812 20:01:52.511169 9355 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:52.721720 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.189s user 0.003s sys 0.035s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:01:52.843616 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.122s user 0.000s sys 0.039s
I20260812 20:01:52.844884 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:53.095458 9337 catalog_manager.cc:1295] Loaded cluster ID: 0833137b24d8420fb4f4a2a536e57642
I20260812 20:01:53.095613 9337 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: loading cluster ID for follower catalog manager: success
I20260812 20:01:53.098395 9337 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: acquiring CA information for follower catalog manager: success
I20260812 20:01:53.100641 9337 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:53.134682 9351 catalog_manager.cc:1295] Loaded cluster ID: 0833137b24d8420fb4f4a2a536e57642
I20260812 20:01:53.134842 9351 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: loading cluster ID for follower catalog manager: success
I20260812 20:01:53.137745 9351 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: acquiring CA information for follower catalog manager: success
I20260812 20:01:53.140008 9351 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 20:01:53.287240 9370 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:53.287742 9370 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:53.287916 9370 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:53.296967 9370 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:53.297230 9370 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:53.310381 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:53.312302 9370 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:53.313939 9370 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:53.324707 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:53.324744 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:53.325824 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:53.326344 9370 server_base.cc:1061] running on GCE node
I20260812 20:01:53.327029 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:53.327247 9370 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:53.327364 9370 hybrid_clock.cc:648] HybridClock initialized: now 1786564913327319 us; error 0 us; skew 500 ppm
I20260812 20:01:53.327697 9370 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:53 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:53.331579 9370 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:53.333750 9370 webserver.cc:533] Webserver started at http://127.25.14.1:44539/ using document root <none> and password file <none>
I20260812 20:01:53.334571 9370 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:53.334681 9370 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:53.335017 9370 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:53.337808 9370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:53.338685 9370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:53.345448 9370 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:01:53.349772 9387 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:53.351639 9370 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:01:53.351903 9370 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:53.352151 9370 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:53.384397 9370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:53.388824 9370 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:53.389168 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:53.390333 9370 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:53.392217 9370 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:53.392338 9370 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:53.392503 9370 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:53.392580 9370 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:53.432199 9370 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34349
I20260812 20:01:53.432296 9518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34349 every 8 connection(s)
I20260812 20:01:53.433787 9370 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:53.442528 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9370
I20260812 20:01:53.442845 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:01:53 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:53 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:53 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:53.482558 9521 heartbeater.cc:344] Connected to a master server at 127.25.14.61:36639
I20260812 20:01:53.482502 9520 heartbeater.cc:344] Connected to a master server at 127.25.14.62:42637
I20260812 20:01:53.483083 9521 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:53.483084 9520 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:53.482460 9519 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43461
I20260812 20:01:53.484078 9519 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:53.484548 9520 heartbeater.cc:507] Master 127.25.14.62:42637 requested a full tablet report, sending...
I20260812 20:01:53.484650 9519 heartbeater.cc:507] Master 127.25.14.60:43461 requested a full tablet report, sending...
I20260812 20:01:53.485836 9521 heartbeater.cc:507] Master 127.25.14.61:36639 requested a full tablet report, sending...
I20260812 20:01:53.487300 9079 ts_manager.cc:194] Registered new tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:53.487622 9252 ts_manager.cc:194] Registered new tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:53.487727 9165 ts_manager.cc:194] Registered new tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:53.489601 9165 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:38421
I20260812 20:01:53.502265 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:01:53.675365 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:42799
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:54.075906 9535 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:54.076355 9535 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:54.076517 9535 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:54.085469 9535 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:54.085687 9535 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:54.085829 9535 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:54.085917 9535 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:54.097702 9535 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:54.099713 9535 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:54.101310 9535 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:54.111611 9541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:54.111605 9545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:54.111560 9540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:54.113708 9535 server_base.cc:1061] running on GCE node
I20260812 20:01:54.114684 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:54.114912 9535 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:54.115042 9535 hybrid_clock.cc:648] HybridClock initialized: now 1786564914114998 us; error 0 us; skew 500 ppm
I20260812 20:01:54.115445 9535 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:54.119706 9535 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:54.122201 9535 webserver.cc:533] Webserver started at http://127.25.14.59:38285/ using document root <none> and password file <none>
I20260812 20:01:54.123070 9535 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:54.123210 9535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:54.123641 9535 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:01:54.126394 9535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
format_stamp: "Formatted at 2026-08-12 20:01:54 on dist-test-slave-p3t5"
I20260812 20:01:54.127280 9535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
format_stamp: "Formatted at 2026-08-12 20:01:54 on dist-test-slave-p3t5"
I20260812 20:01:54.133792 9535 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:01:54.137739 9551 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:54.139326 9535 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:01:54.139533 9535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
format_stamp: "Formatted at 2026-08-12 20:01:54 on dist-test-slave-p3t5"
I20260812 20:01:54.139758 9535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:54.162730 9535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:54.167418 9535 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:54.167754 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:54.190465 9535 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:42799
I20260812 20:01:54.190492 9611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:42799 every 8 connection(s)
I20260812 20:01:54.192209 9535 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:54.197336 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:01:54.198357 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9535
I20260812 20:01:54.206983 9612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:54.228308 9612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:01:54.242316 9612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:01:54.256642 9612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:54.275255 9612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Bootstrap starting.
I20260812 20:01:54.279048 9612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Neither blocks nor log segments found. Creating new log.
I20260812 20:01:54.280314 9612 log.cc:826] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:54.283535 9612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: No bootstrap required, opened a new log
I20260812 20:01:54.289103 9612 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } }
I20260812 20:01:54.289502 9612 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:01:54.289650 9612 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c799a5da18b49b8ad55ed383f24c9e3, State: Initialized, Role: FOLLOWER
I20260812 20:01:54.290367 9612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } }
I20260812 20:01:54.291412 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } } }
I20260812 20:01:54.291776 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:54.292634 9612 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:01:54.303776 9637 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:01:54.303972 9637 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:01:54.314059 9535 master_runner.cc:429] Detected that this master 1c799a5da18b49b8ad55ed383f24c9e3 is joining an existing cluster
I20260812 20:01:54.314184 9535 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:01:54.315407 9535 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:01:54.315835 9535 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:01:54.316056 9535 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: stopping tablet replica
I20260812 20:01:54.318069 9535 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:42799
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564911, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:54.393824 9165 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:42799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 42799
}
attrs {
promote: true
}
}
dest_uuid: "def8d6607a144030aa1972bf4a1b39ff"
cas_config_opid_index: -1
I20260812 20:01:54.395737 9165 consensus_queue.cc:237] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } }
I20260812 20:01:54.399039 9090 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:01:54.399751 9308 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:54.400265 9262 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:01:54.401005 9150 consensus_peers.cc:597] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff -> Peer 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59:42799): Couldn't send request to peer 1c799a5da18b49b8ad55ed383f24c9e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:01:54.401027 9308 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:01:54.404111 9308 raft_consensus.cc:2955] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.405299 9262 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.406543 9090 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.408659 9308 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:42799
I20260812 20:01:54.412163 9535 master.cc:562] Master@127.25.14.59:42799 shutting down...
I20260812 20:01:54.415746 9658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.416101 9658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:54.420575 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.420931 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:54.431484 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.431820 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:54.436062 9535 master.cc:584] Master@127.25.14.59:42799 shutdown complete.
I20260812 20:01:54.436143 9535 master_runner.cc:306] Clearing existing system tablet
I20260812 20:01:54.436545 9535 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:54.436656 9535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:54.437311 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.437451 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.438243 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:54.438292 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:54.439458 9535 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260812 20:01:54.440939 9663 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:54.441295 9535 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:54.441568 9535 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 20:01:54.441682 9535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
format_stamp: "Formatted at 2026-08-12 20:01:54 on dist-test-slave-p3t5"
I20260812 20:01:54.445176 9535 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:01:54.450538 9535 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:01:54.450791 9535 log.cc:1199] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:01:54.451272 9535 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Deleting consensus metadata
I20260812 20:01:54.451545 9535 master_runner.cc:316] Copying system tablet from 127.25.14.61:36639
I20260812 20:01:54.454493 9535 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.61:36639
I20260812 20:01:54.468981 9188 tablet_copy_service.cc:140] P def8d6607a144030aa1972bf4a1b39ff: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c799a5da18b49b8ad55ed383f24c9e3 ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:45914)
I20260812 20:01:54.469398 9188 tablet_copy_service.cc:161] P def8d6607a144030aa1972bf4a1b39ff: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c799a5da18b49b8ad55ed383f24c9e3 at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:45914: session id = 1c799a5da18b49b8ad55ed383f24c9e3-00000000000000000000000000000000
I20260812 20:01:54.474192 9188 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:01:54.476912 9535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:01:54.482481 9535 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: tablet copy: Starting download of 0 data blocks...
I20260812 20:01:54.482889 9535 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: tablet copy: Starting download of 1 WAL segments...
I20260812 20:01:54.486279 9535 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:01:54.490142 9188 tablet_copy_service.cc:342] P def8d6607a144030aa1972bf4a1b39ff: Request end of tablet copy session 1c799a5da18b49b8ad55ed383f24c9e3-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:45914
I20260812 20:01:54.490581 9188 tablet_copy_service.cc:434] P def8d6607a144030aa1972bf4a1b39ff: ending tablet copy session 1c799a5da18b49b8ad55ed383f24c9e3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c799a5da18b49b8ad55ed383f24c9e3
I20260812 20:01:54.492369 9521 heartbeater.cc:499] Master 127.25.14.61:36639 was elected leader, sending a full tablet report...
I20260812 20:01:54.504031 9535 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:54.506953 9671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:54.506979 9672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:54.508862 9674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:54.509892 9535 server_base.cc:1061] running on GCE node
I20260812 20:01:54.510236 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:54.510308 9535 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:54.510402 9535 hybrid_clock.cc:648] HybridClock initialized: now 1786564914510362 us; error 0 us; skew 500 ppm
I20260812 20:01:54.510632 9535 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:54.512773 9535 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:01:54.514156 9535 webserver.cc:533] Webserver started at http://127.25.14.59:46267/ using document root <none> and password file <none>
I20260812 20:01:54.514515 9535 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:54.514616 9535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:54.517057 9535 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 20:01:54.519189 9681 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:54.519748 9535 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 20:01:54.519906 9535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "1c799a5da18b49b8ad55ed383f24c9e3"
format_stamp: "Formatted at 2026-08-12 20:01:54 on dist-test-slave-p3t5"
I20260812 20:01:54.520087 9535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:54.542102 9535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:54.546018 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:54.575917 9747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:42799 every 8 connection(s)
I20260812 20:01:54.575911 9535 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:42799
I20260812 20:01:54.577193 9535 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:01:54.580865 9748 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:54.583603 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Bootstrap starting.
I20260812 20:01:54.601595 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:54.602277 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: Bootstrap complete.
I20260812 20:01:54.603593 9748 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } }
I20260812 20:01:54.603856 9748 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c799a5da18b49b8ad55ed383f24c9e3, State: Initialized, Role: LEARNER
I20260812 20:01:54.604159 9748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } }
I20260812 20:01:54.605346 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.605690 9748 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:54.605705 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:01:54.615702 9765 catalog_manager.cc:1295] Loaded cluster ID: 0833137b24d8420fb4f4a2a536e57642
I20260812 20:01:54.615813 9765 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: loading cluster ID for follower catalog manager: success
I20260812 20:01:54.618615 9765 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: acquiring CA information for follower catalog manager: success
I20260812 20:01:54.620829 9765 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:01:54.660516 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9370
Aug 12 20:01:54 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:01:54.679473 9513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:54.790949 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9370
I20260812 20:01:54.824818 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34349
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--webserver_port=44539
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:54.832594 9717 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:01:54.833557 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def8d6607a144030aa1972bf4a1b39ff. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:54.833923 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:01:55.225566 9768 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:55.226084 9768 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:55.226239 9768 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:55.240666 9768 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:55.240991 9768 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:55.253111 9768 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44539
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:55.254957 9768 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:55.256523 9768 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:55.268676 9776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:55.268626 9775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:55.269379 9778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:55.270176 9768 server_base.cc:1061] running on GCE node
I20260812 20:01:55.270911 9768 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:55.271134 9768 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:55.271270 9768 hybrid_clock.cc:648] HybridClock initialized: now 1786564915271242 us; error 0 us; skew 500 ppm
I20260812 20:01:55.271648 9768 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:55 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:55.275679 9768 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:55.277814 9768 webserver.cc:533] Webserver started at http://127.25.14.1:44539/ using document root <none> and password file <none>
I20260812 20:01:55.278684 9768 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:55.278827 9768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:55.285461 9768 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 20:01:55.288964 9784 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:55.290630 9768 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:55.290901 9768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:55.291841 9768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:55.354228 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: true } } }
I20260812 20:01:55.354542 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:55.355782 9768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:55.356209 9792 raft_consensus.cc:1064] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff: attempting to promote NON_VOTER 1c799a5da18b49b8ad55ed383f24c9e3 to VOTER
I20260812 20:01:55.357106 9792 consensus_queue.cc:237] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } }
I20260812 20:01:55.359932 9262 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:55.359975 9090 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:55.360719 9654 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:55.361089 9790 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:55.361245 9717 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 1 LEARNER]: Refusing update from remote peer def8d6607a144030aa1972bf4a1b39ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:01:55.361964 9790 consensus_queue.cc:1048] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:01:55.362462 9768 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:55.362812 9768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:55.364367 9768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:55.364557 9654 raft_consensus.cc:2955] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.365979 9090 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.370020 9799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.370249 9799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:55.370230 9768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:55.370374 9768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:55.370559 9768 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:55.370692 9768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:55.370772 9790 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.370380 9717 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.371090 9790 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:55.371346 9262 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c799a5da18b49b8ad55ed383f24c9e3 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.371592 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.371879 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:55.372169 9790 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.372386 9790 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:55.372633 9794 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.372893 9794 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:55.378225 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.378516 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481bdb5bb5344312be9764d8ff127d5a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:55.379242 9802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:55.379606 9802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c799a5da18b49b8ad55ed383f24c9e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:55.419654 9768 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34349
I20260812 20:01:55.419705 9927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34349 every 8 connection(s)
I20260812 20:01:55.421515 9768 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:55.431113 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9768
Aug 12 20:01:55 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:55 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:55 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:01:55 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:55.472568 9928 heartbeater.cc:344] Connected to a master server at 127.25.14.59:42799
I20260812 20:01:55.473034 9928 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:55.473219 9931 heartbeater.cc:344] Connected to a master server at 127.25.14.61:36639
I20260812 20:01:55.473390 9931 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:55.474093 9929 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43461
I20260812 20:01:55.474303 9929 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:55.474606 9931 heartbeater.cc:507] Master 127.25.14.61:36639 requested a full tablet report, sending...
I20260812 20:01:55.474915 9929 heartbeater.cc:507] Master 127.25.14.60:43461 requested a full tablet report, sending...
I20260812 20:01:55.475697 9930 heartbeater.cc:344] Connected to a master server at 127.25.14.62:42637
I20260812 20:01:55.475793 9928 heartbeater.cc:507] Master 127.25.14.59:42799 requested a full tablet report, sending...
I20260812 20:01:55.475900 9930 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:55.476451 9930 heartbeater.cc:507] Master 127.25.14.62:42637 requested a full tablet report, sending...
I20260812 20:01:55.477283 9252 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:55.477728 9706 ts_manager.cc:194] Registered new tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:55.478492 9165 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:55.478801 9079 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:55.479650 9165 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:34709
I20260812 20:01:55.488304 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:55.497610 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:01:55.497865 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9040
I20260812 20:01:55.498986 9114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:55.622864 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9040
I20260812 20:01:55.651683 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:42637
--webserver_interface=127.25.14.62
--webserver_port=37979
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:55.959663 9149 consensus_peers.cc:597] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff -> Peer 524da588370e4a3bbc1da01a524832c4 (127.25.14.62:42637): Couldn't send request to peer 524da588370e4a3bbc1da01a524832c4. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:42637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:01:56.048244 9946 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:56.048648 9946 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:56.048818 9946 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:56.057814 9946 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:01:56.057945 9946 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:56.058051 9946 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:01:56.058136 9946 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:01:56.070269 9946 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=37979
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:56.072206 9946 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:56.073786 9946 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:56.082880 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:56.082856 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:56.085952 9946 server_base.cc:1061] running on GCE node
W20260812 20:01:56.087100 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:56.088171 9946 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:56.088348 9946 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:56.088426 9946 hybrid_clock.cc:648] HybridClock initialized: now 1786564916088406 us; error 0 us; skew 500 ppm
I20260812 20:01:56.088738 9946 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:56.092736 9946 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:01:56.095022 9946 webserver.cc:533] Webserver started at http://127.25.14.62:37979/ using document root <none> and password file <none>
I20260812 20:01:56.095780 9946 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:56.095901 9946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:56.102044 9946 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:01:56.105213 9964 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:56.106832 9946 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:01:56.107051 9946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "524da588370e4a3bbc1da01a524832c4"
format_stamp: "Formatted at 2026-08-12 20:01:50 on dist-test-slave-p3t5"
I20260812 20:01:56.107951 9946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:56.126268 9946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:56.130831 9946 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:56.131155 9946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:56.153693 9946 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:42637
I20260812 20:01:56.153677 10021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:42637 every 8 connection(s)
I20260812 20:01:56.155596 9946 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:01:56.163532 10022 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:01:56.164136 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 9946
I20260812 20:01:56.164472 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9768
I20260812 20:01:56.168454 10022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Bootstrap starting.
I20260812 20:01:56.183722 9922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:56.191637 10022 log.cc:826] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Log is configured to *not* fsync() on all Append() calls
I20260812 20:01:56.203357 10022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:01:56.204046 10022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: Bootstrap complete.
I20260812 20:01:56.212154 10022 raft_consensus.cc:359] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } }
I20260812 20:01:56.212989 10022 raft_consensus.cc:740] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524da588370e4a3bbc1da01a524832c4, State: Initialized, Role: FOLLOWER
I20260812 20:01:56.214035 10022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } }
I20260812 20:01:56.215046 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:56.215420 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:01:56.216373 10022 sys_catalog.cc:565] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:01:56.246272 10045 catalog_manager.cc:1295] Loaded cluster ID: 0833137b24d8420fb4f4a2a536e57642
I20260812 20:01:56.246465 10045 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: loading cluster ID for follower catalog manager: success
I20260812 20:01:56.251063 10045 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: acquiring CA information for follower catalog manager: success
I20260812 20:01:56.254323 10045 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:01:56.303525 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9768
I20260812 20:01:56.335623 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34349
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--webserver_port=44539
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:01:56.409317 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:56.409654 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:56.421211 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def8d6607a144030aa1972bf4a1b39ff. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:56.421583 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524da588370e4a3bbc1da01a524832c4 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:01:56.750030 10053 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:56.750361 10053 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:56.750471 10053 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:56.761540 10053 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:56.761945 10053 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:56.778955 10053 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44539
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:56.781404 10053 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:56.783180 10053 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:56.798143 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:56.798072 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:56.798103 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:56.799000 10053 server_base.cc:1061] running on GCE node
I20260812 20:01:56.799714 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:56.799923 10053 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:56.800041 10053 hybrid_clock.cc:648] HybridClock initialized: now 1786564916799999 us; error 0 us; skew 500 ppm
I20260812 20:01:56.800391 10053 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:56.805337 10053 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:56.808091 10053 webserver.cc:533] Webserver started at http://127.25.14.1:44539/ using document root <none> and password file <none>
I20260812 20:01:56.809245 10053 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:56.809433 10053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:56.818095 10053 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 20:01:56.818013 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "def8d6607a144030aa1972bf4a1b39ff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524da588370e4a3bbc1da01a524832c4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 42637 } } peers { permanent_uuid: "def8d6607a144030aa1972bf4a1b39ff" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 36639 } } peers { permanent_uuid: "481bdb5bb5344312be9764d8ff127d5a" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 43461 } } peers { permanent_uuid: "1c799a5da18b49b8ad55ed383f24c9e3" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 42799 } attrs { promote: false } } }
I20260812 20:01:56.818285 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P def8d6607a144030aa1972bf4a1b39ff [sys.catalog]: This master's current role is: LEADER
I20260812 20:01:56.822620 10074 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:56.824473 10053 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 20:01:56.824671 10053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:56.825495 10053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:56.843405 10053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:56.850607 10053 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:56.851003 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:56.852759 10053 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:56.855342 10053 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:56.855504 10053 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:56.855633 10053 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:56.855733 10053 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:56.903896 10053 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34349
I20260812 20:01:56.903990 10209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34349 every 8 connection(s)
I20260812 20:01:56.905611 10053 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:56.910122 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10053
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:56 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:56.975008 10212 heartbeater.cc:344] Connected to a master server at 127.25.14.62:42637
I20260812 20:01:56.975518 10212 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:56.976804 10212 heartbeater.cc:507] Master 127.25.14.62:42637 requested a full tablet report, sending...
I20260812 20:01:56.977990 10211 heartbeater.cc:344] Connected to a master server at 127.25.14.60:43461
I20260812 20:01:56.978160 10211 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:56.978394 10214 heartbeater.cc:344] Connected to a master server at 127.25.14.61:36639
I20260812 20:01:56.978567 10214 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:56.978859 10211 heartbeater.cc:507] Master 127.25.14.60:43461 requested a full tablet report, sending...
I20260812 20:01:56.979215 10214 heartbeater.cc:507] Master 127.25.14.61:36639 requested a full tablet report, sending...
I20260812 20:01:56.980358 9252 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:56.981299 9982 ts_manager.cc:194] Registered new tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:56.983215 9165 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:56.984381 9165 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:40821
I20260812 20:01:56.986647 10210 heartbeater.cc:344] Connected to a master server at 127.25.14.59:42799
I20260812 20:01:56.986861 10210 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:56.987213 10210 heartbeater.cc:507] Master 127.25.14.59:42799 requested a full tablet report, sending...
I20260812 20:01:56.988869 9706 ts_manager.cc:194] Re-registered known tserver with Master: 3a841349ef9f4f398083d5be5f71cb98 (127.25.14.1:34349)
I20260812 20:01:57.132293 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10053
W20260812 20:01:57.133916 10229 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:57.129833 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.25.14.1:34349)
0812 20:01:57.130523 (+ 690us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:57.130535 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:57.130579 (+ 44us) server_negotiation.cc:408] Connection header received
0812 20:01:57.130666 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:57.130674 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:57.130909 (+ 235us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:57.131130 (+ 221us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:57.132267 (+ 1137us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49928: BlockingRecv error: recv got EOF from 127.0.0.1:49928 (error 108)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":336,"threads_started":1}
W20260812 20:01:57.135694 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10053: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34349: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:57.135831 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10053
W20260812 20:01:57.135910 10232 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:57.133652 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.25.14.1:34349)
0812 20:01:57.134388 (+ 736us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:57.134392 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:57.134410 (+ 18us) server_negotiation.cc:408] Connection header received
0812 20:01:57.134488 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:57.134494 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:57.134570 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:57.134699 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:57.135617 (+ 918us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49934: BlockingRecv error: recv got EOF from 127.0.0.1:49934 (error 108)
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":487,"threads_started":1}
I20260812 20:01:57.163324 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34349
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--webserver_port=44539
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:57.559779 10233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:57.560171 10233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:57.560376 10233 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:01:57.560472 10233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:57.569336 10233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:57.569783 10233 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:57.583916 10233 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44539
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:57.585769 10233 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:57.587340 10233 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:57.598157 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:57.598677 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:57.599217 10242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:57.600158 10233 server_base.cc:1061] running on GCE node
I20260812 20:01:57.600821 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:57.600970 10233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:57.601055 10233 hybrid_clock.cc:648] HybridClock initialized: now 1786564917601019 us; error 0 us; skew 500 ppm
I20260812 20:01:57.601310 10233 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:57 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564917, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:57.604990 10233 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:57.606889 10233 webserver.cc:533] Webserver started at http://127.25.14.1:44539/ using document root <none> and password file <none>
I20260812 20:01:57.607776 10233 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:57.607903 10233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:57.614041 10233 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:01:57.617094 10249 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:57.618744 10233 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:01:57.618942 10233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:57.619664 10233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:57.648793 10233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:57.653501 10233 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:57.653854 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:57.654945 10233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:57.656706 10233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:57.656819 10233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:57.656985 10233 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:57.657065 10233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:01:57.657508 10233 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:01:57.657606 10233 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:01:57.657729 10233 fault_injection.cc:59] injecting fault for kudu (pid 10233): FLAGS_fault_before_start (process will exit)
I20260812 20:01:57.686561 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34349
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461
--webserver_port=44539
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:01:58.075714 10259 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:01:58.076146 10259 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:01:58.076310 10259 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:01:58.085157 10259 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:01:58.085433 10259 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:01:58.096800 10259 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=44539
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:42637,127.25.14.61:36639,127.25.14.60:43461,127.25.14.59:42799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:01:58.098762 10259 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:01:58.100330 10259 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:01:58.110747 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:58.110778 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:01:58.111574 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:01:58.112424 10259 server_base.cc:1061] running on GCE node
I20260812 20:01:58.113137 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 20:01:58.113402 10259 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 20:01:58.113533 10259 hybrid_clock.cc:648] HybridClock initialized: now 1786564918113501 us; error 0 us; skew 500 ppm
I20260812 20:01:58.113938 10259 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:01:58 dist-test-slave-p3t5 krb5kdc[8679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:01:58.118023 10259 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:01:58.120146 10259 webserver.cc:533] Webserver started at http://127.25.14.1:44539/ using document root <none> and password file <none>
I20260812 20:01:58.121042 10259 fs_manager.cc:362] Metadata directory not provided
I20260812 20:01:58.121199 10259 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:01:58.127388 10259 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 20:01:58.130499 10276 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:58.132048 10259 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:01:58.132253 10259 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3a841349ef9f4f398083d5be5f71cb98"
format_stamp: "Formatted at 2026-08-12 20:01:53 on dist-test-slave-p3t5"
I20260812 20:01:58.133123 10259 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:01:58.152186 10259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:01:58.156916 10259 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:01:58.157315 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:01:58.158499 10259 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:01:58.160298 10259 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:01:58.160440 10259 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:58.160614 10259 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:01:58.160714 10259 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:01:58.200276 10259 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34349
I20260812 20:01:58.200341 10413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34349 every 8 connection(s)
I20260812 20:01:58.202235 10259 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:01:58.208674 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10259
I20260812 20:01:58.209079 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10259
Aug 12 20:01:58 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:01:58 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:01:58 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:01:58 dist-test-slave-p3t5 krb5kdc[8679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:01:58.267290 10416 heartbeater.cc:344] Connected to a master server at 127.25.14.62:42637
W20260812 20:01:58.267551 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10259: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34349: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:58.267700 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10259
I20260812 20:01:58.267748 10416 heartbeater.cc:461] Registering TS with master...
I20260812 20:01:58.295177 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9946
W20260812 20:01:58.296288 10429 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:58.230559 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58075 (local address 127.25.14.61:36639)
0812 20:01:58.246032 (+ 15473us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:58.246038 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:58.246061 (+ 23us) server_negotiation.cc:408] Connection header received
0812 20:01:58.246127 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:58.246133 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:58.246220 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:58.246341 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:58.251955 (+ 5614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.253852 (+ 1897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:58.256990 (+ 3138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.257532 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:58.264184 (+ 6652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:01:58.264246 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:01:58.264252 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 20:01:58.264297 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:01:58.273576 (+ 9279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:58.295156 (+ 21580us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58075: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":13475,"thread_start_us":15283,"threads_started":1}
W20260812 20:01:58.296304 10421 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:58.206675 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:39271 (local address 127.25.14.59:42799)
0812 20:01:58.207425 (+ 750us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:58.207435 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:58.229193 (+ 21758us) server_negotiation.cc:408] Connection header received
0812 20:01:58.238451 (+ 9258us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:58.238460 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:58.238631 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:58.238757 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:58.244999 (+ 6242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.247049 (+ 2050us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:58.255944 (+ 8895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.256676 (+ 732us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:58.263673 (+ 6997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:01:58.263731 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:01:58.263740 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 20:01:58.263804 (+ 64us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:01:58.272138 (+ 8334us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:58.295248 (+ 23110us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:39271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":371,"threads_started":1}
W20260812 20:01:58.297231 10420 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:01:58.206660 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58463 (local address 127.25.14.60:43461)
0812 20:01:58.207727 (+ 1067us) server_negotiation.cc:207] Beginning negotiation
0812 20:01:58.207734 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 20:01:58.236330 (+ 28596us) server_negotiation.cc:408] Connection header received
0812 20:01:58.236561 (+ 231us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:01:58.236572 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:01:58.236700 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:01:58.236877 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:01:58.244239 (+ 7362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.245720 (+ 1481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:01:58.249490 (+ 3770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:01:58.250062 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:01:58.259069 (+ 9007us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0812 20:01:58.259125 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0812 20:01:58.259129 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0812 20:01:58.259168 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0812 20:01:58.263881 (+ 4713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:58.265057 (+ 1176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0812 20:01:58.265068 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0812 20:01:58.265073 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0812 20:01:58.265194 (+ 121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0812 20:01:58.296275 (+ 31081us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58463: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":841,"threads_started":1}
W20260812 20:01:58.298161 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9946: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:42637: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:01:58.298350 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9946
I20260812 20:01:58.327440 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9130
I20260812 20:01:58.328884 9200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:58.452013 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9130
W20260812 20:01:58.481192 9236 connection.cc:573] server connection from 127.0.0.1:49894 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:01:58.481670 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9217
I20260812 20:01:58.483068 9285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:58.605175 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9217
I20260812 20:01:58.635464 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 9535
I20260812 20:01:58.636826 9743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:01:58.753419 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 9535
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/9 (28633 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/10
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:01:59 dist-test-slave-p3t5 krb5kdc[10444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:01:59 dist-test-slave-p3t5 krb5kdc[10444](info): set up 2 sockets
Aug 12 20:01:59 dist-test-slave-p3t5 krb5kdc[10444](info): commencing operation
krb5kdc: starting...
W20260812 20:02:01.526474 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.324s user 0.004s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:02:01 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:02:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:01Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 20:02:09: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 20:02:12 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564932, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:02:18 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564938, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:02:18 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564938, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:21.165127 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:37229
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:21.566663 10803 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:21.567041 10803 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:21.567129 10803 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:21.576052 10803 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:21.576171 10803 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:21.576228 10803 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:21.576292 10803 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:21.587785 10803 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:21.589545 10803 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:21.591167 10803 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:21.600781 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:21.600843 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:21.601234 10814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:21.602053 10803 server_base.cc:1061] running on GCE node
I20260812 20:02:21.603040 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:21.604611 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:21.605895 10803 hybrid_clock.cc:648] HybridClock initialized: now 1786564941605832 us; error 69 us; skew 500 ppm
I20260812 20:02:21.606397 10803 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:21 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564941, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:21.610221 10803 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:02:21.612241 10803 webserver.cc:533] Webserver started at http://127.25.14.62:35551/ using document root <none> and password file <none>
I20260812 20:02:21.613149 10803 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:21.613315 10803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:21.613777 10803 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:21.616612 10803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "5cdb87ac3fd549dfb13780881f5015b1"
format_stamp: "Formatted at 2026-08-12 20:02:21 on dist-test-slave-p3t5"
I20260812 20:02:21.617548 10803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "5cdb87ac3fd549dfb13780881f5015b1"
format_stamp: "Formatted at 2026-08-12 20:02:21 on dist-test-slave-p3t5"
I20260812 20:02:21.623667 10803 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260812 20:02:21.627688 10823 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:21.629295 10803 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 20:02:21.629521 10803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5cdb87ac3fd549dfb13780881f5015b1"
format_stamp: "Formatted at 2026-08-12 20:02:21 on dist-test-slave-p3t5"
I20260812 20:02:21.629825 10803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:21.647790 10803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:21.652503 10803 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:21.652848 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:21.675625 10803 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:37229
I20260812 20:02:21.675778 10892 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:37229 every 8 connection(s)
I20260812 20:02:21.677359 10803 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:02:21.682214 10893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:21.687342 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10803
I20260812 20:02:21.687584 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:02:21.688786 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:21 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564941, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:21.713598 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:02:21.714900 10824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
W20260812 20:02:21.716694 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
I20260812 20:02:21.766175 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 1
W20260812 20:02:21.768455 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:02:21.836737 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 2
W20260812 20:02:21.839035 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:21.860811 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45939
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:21.930325 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 3
W20260812 20:02:21.932737 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
I20260812 20:02:22.076052 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 4
W20260812 20:02:22.078526 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
W20260812 20:02:22.254832 10906 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:22.255249 10906 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:22.255419 10906 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:22.264459 10906 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:22.264592 10906 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:22.264683 10906 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:22.264772 10906 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:22.276403 10906 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:22.278453 10906 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:22.280081 10906 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:22.290994 10917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:22.291062 10912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:22.291078 10915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:22.292469 10906 server_base.cc:1061] running on GCE node
I20260812 20:02:22.293447 10906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:22.295200 10906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:22.296453 10906 hybrid_clock.cc:648] HybridClock initialized: now 1786564942296392 us; error 48 us; skew 500 ppm
I20260812 20:02:22.296936 10906 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:22 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:22.300565 10906 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:02:22.302618 10906 webserver.cc:533] Webserver started at http://127.25.14.61:34007/ using document root <none> and password file <none>
I20260812 20:02:22.303438 10906 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:22.303547 10906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:22.303908 10906 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:22.306757 10906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "2d922e9089d44bbebf7129a0ffade3ce"
format_stamp: "Formatted at 2026-08-12 20:02:22 on dist-test-slave-p3t5"
I20260812 20:02:22.307622 10906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "2d922e9089d44bbebf7129a0ffade3ce"
format_stamp: "Formatted at 2026-08-12 20:02:22 on dist-test-slave-p3t5"
I20260812 20:02:22.313731 10906 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 20:02:22.317601 10926 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:22.319221 10906 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 20:02:22.319432 10906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "2d922e9089d44bbebf7129a0ffade3ce"
format_stamp: "Formatted at 2026-08-12 20:02:22 on dist-test-slave-p3t5"
I20260812 20:02:22.319662 10906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:22.363941 10906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:22.373129 10906 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:22.373485 10906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:22.377833 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 5
W20260812 20:02:22.380061 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:45939: Network error: Client connection negotiation failed: client connection to 127.25.14.61:45939: connect: Connection refused (error 111)
I20260812 20:02:22.397509 10985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:45939 every 8 connection(s)
I20260812 20:02:22.397523 10906 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:45939
I20260812 20:02:22.399359 10906 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:02:22.404343 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10906
I20260812 20:02:22.404587 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:02:22.404881 10988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:22.414855 10988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:22 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:22.440850 10988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 20:02:22 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:22.459852 10988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:02:22.461229 10930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
W20260812 20:02:22.463186 10988 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:22.512590 10988 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 1
W20260812 20:02:22.514925 10988 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:02:22.583185 10988 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 2
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:02:22.585546 10988 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
I20260812 20:02:22.586129 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35463
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:22.676839 10988 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 3
W20260812 20:02:22.679565 10988 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
I20260812 20:02:22.822919 10988 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 4
W20260812 20:02:22.825690 10988 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
I20260812 20:02:22.927326 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } attempt: 6
Aug 12 20:02:22 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564941, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:22.943532 10893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:02:22.946096 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
W20260812 20:02:22.985059 10999 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:22.985466 10999 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:22.985674 10999 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:22.994483 10999 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:22.994608 10999 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:22.994726 10999 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:22.994807 10999 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:22.998360 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 1
W20260812 20:02:23.000607 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
I20260812 20:02:23.007611 10999 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:23.009418 10999 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:23.011006 10999 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:23.021090 11010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:23.021190 11009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:23.021857 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:23.022802 10999 server_base.cc:1061] running on GCE node
I20260812 20:02:23.023859 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:23.025701 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:23.026942 10999 hybrid_clock.cc:648] HybridClock initialized: now 1786564943026888 us; error 41 us; skew 500 ppm
I20260812 20:02:23.027390 10999 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:23.031260 10999 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:02:23.033174 10999 webserver.cc:533] Webserver started at http://127.25.14.60:44043/ using document root <none> and password file <none>
I20260812 20:02:23.034094 10999 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:23.034204 10999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:23.034603 10999 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:23.037423 10999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "34a60c59e1f042b9bc12abe767959159"
format_stamp: "Formatted at 2026-08-12 20:02:23 on dist-test-slave-p3t5"
I20260812 20:02:23.038388 10999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "34a60c59e1f042b9bc12abe767959159"
format_stamp: "Formatted at 2026-08-12 20:02:23 on dist-test-slave-p3t5"
I20260812 20:02:23.044811 10999 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:02:23.049052 11021 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:23.051136 10999 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:23.051396 10999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "34a60c59e1f042b9bc12abe767959159"
format_stamp: "Formatted at 2026-08-12 20:02:23 on dist-test-slave-p3t5"
I20260812 20:02:23.051661 10999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:23.074898 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 2
W20260812 20:02:23.077271 10893 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:35463: Network error: Client connection negotiation failed: client connection to 127.25.14.60:35463: connect: Connection refused (error 111)
I20260812 20:02:23.084585 10999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:23.089757 10999 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:23.090127 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:23.115886 10999 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:35463
I20260812 20:02:23.115933 11082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:35463 every 8 connection(s)
I20260812 20:02:23.117672 10999 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:02:23.119185 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 10999
I20260812 20:02:23.119427 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:02:23.124274 11083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:23.125133 10988 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 5
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:23.135028 11083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:23.159204 10988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: Bootstrap starting.
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:23.164034 10988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:23.165642 10988 log.cc:826] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:23.170150 11083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:23.170578 10988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: No bootstrap required, opened a new log
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:23.178119 10988 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.178614 10988 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:23.178862 10988 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d922e9089d44bbebf7129a0ffade3ce, State: Initialized, Role: FOLLOWER
I20260812 20:02:23.179750 10988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:23.181111 11093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.181596 11093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.182616 10988 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:23.190397 11083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:23.190548 10893 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } attempt: 3
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:23 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564941, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:23.218503 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Bootstrap starting.
I20260812 20:02:23.221100 11083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Bootstrap starting.
I20260812 20:02:23.223451 10893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:23.214892 10906 master_runner.cc:187] Error getting master registration for 127.25.14.62:37229: OK, instance_id { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" instance_seqno: 1786564941669243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:02:23.224274 10906 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:02:23.225229 10893 log.cc:826] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Log is configured to *not* fsync() on all Append() calls
W20260812 20:02:23.227541 11107 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:23.227783 11107 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:23.230965 11083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:23.232581 11083 log.cc:826] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:23.232894 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: No bootstrap required, opened a new log
I20260812 20:02:23.236202 11083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: No bootstrap required, opened a new log
I20260812 20:02:23.241907 10893 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.242359 10893 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:23.242545 10893 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb87ac3fd549dfb13780881f5015b1, State: Initialized, Role: FOLLOWER
I20260812 20:02:23.243584 10893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.244797 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.245218 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.246352 10893 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:23.246409 11083 raft_consensus.cc:359] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.246860 11083 raft_consensus.cc:385] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:23.247035 11083 raft_consensus.cc:740] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a60c59e1f042b9bc12abe767959159, State: Initialized, Role: FOLLOWER
I20260812 20:02:23.247834 11083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.255198 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.257911 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.258379 11083 sys_catalog.cc:565] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:02:23.281126 11124 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:23.281370 11124 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:23.272550 10803 master_runner.cc:187] Error getting master registration for 127.25.14.60:35463: OK, instance_id { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" instance_seqno: 1786564943109043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:02:23.282936 10803 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:02:23.284685 11115 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:02:23.285061 11115 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.287587 11115 leader_election.cc:290] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cdb87ac3fd549dfb13780881f5015b1 (127.25.14.62:37229), 2d922e9089d44bbebf7129a0ffade3ce (127.25.14.61:45939)
I20260812 20:02:23.290201 10863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a60c59e1f042b9bc12abe767959159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb87ac3fd549dfb13780881f5015b1" is_pre_election: true
I20260812 20:02:23.290928 10863 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a60c59e1f042b9bc12abe767959159 in term 0.
I20260812 20:02:23.291885 10958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a60c59e1f042b9bc12abe767959159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d922e9089d44bbebf7129a0ffade3ce" is_pre_election: true
I20260812 20:02:23.292514 10958 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a60c59e1f042b9bc12abe767959159 in term 0.
I20260812 20:02:23.295543 11023 leader_election.cc:304] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d922e9089d44bbebf7129a0ffade3ce, 34a60c59e1f042b9bc12abe767959159; no voters:
I20260812 20:02:23.297639 11115 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:02:23.298061 11115 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:02:23.298296 11115 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:02:23.303300 11115 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.305469 10863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a60c59e1f042b9bc12abe767959159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdb87ac3fd549dfb13780881f5015b1"
I20260812 20:02:23.305845 10863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:02:23.311065 10863 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a60c59e1f042b9bc12abe767959159 in term 1.
I20260812 20:02:23.317821 11023 leader_election.cc:304] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a60c59e1f042b9bc12abe767959159, 5cdb87ac3fd549dfb13780881f5015b1; no voters:
I20260812 20:02:23.318348 11115 leader_election.cc:290] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [CANDIDATE]: Term 1 election: Requested vote from peers 5cdb87ac3fd549dfb13780881f5015b1 (127.25.14.62:37229), 2d922e9089d44bbebf7129a0ffade3ce (127.25.14.61:45939)
I20260812 20:02:23.318701 11115 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:02:23.318909 11115 raft_consensus.cc:697] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 1 LEADER]: Becoming Leader. State: Replica: 34a60c59e1f042b9bc12abe767959159, State: Running, Role: LEADER
I20260812 20:02:23.319685 11115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } }
I20260812 20:02:23.328132 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a60c59e1f042b9bc12abe767959159. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.328495 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:23.333936 10958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a60c59e1f042b9bc12abe767959159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d922e9089d44bbebf7129a0ffade3ce"
I20260812 20:02:23.334268 10958 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:02:23.342852 11028 tablet.cc:2406] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:02:23.345656 10958 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a60c59e1f042b9bc12abe767959159 in term 1.
I20260812 20:02:23.350611 11140 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:02:23.369269 11140 catalog_manager.cc:1520] Initializing Kudu cluster ID...
W20260812 20:02:23.376751 10932 tablet.cc:2406] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:02:23.378640 10958 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:02:23.378710 10863 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:02:23.379906 11115 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:02:23.380179 11136 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 20:02:23.406674 10830 tablet.cc:2406] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:02:23.416517 11093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a60c59e1f042b9bc12abe767959159. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.416827 11093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.420892 11143 mvcc.cc:204] Tried to move back new op lower bound from 7317770008064815104 to 7317770007846612992. Current Snapshot: MvccSnapshot[applied={T|T < 7317770008064815104}]
I20260812 20:02:23.424846 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a60c59e1f042b9bc12abe767959159. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.424810 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.425135 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.425228 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:23.431485 11145 mvcc.cc:204] Tried to move back new op lower bound from 7317770008064815104 to 7317770007846612992. Current Snapshot: MvccSnapshot[applied={T|T < 7317770008064815104}]
I20260812 20:02:23.434500 11140 catalog_manager.cc:1383] Generated new cluster ID: 11d9abb606164b81a8f4c8810a2d6849
I20260812 20:02:23.434626 11140 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:02:23.438416 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.440089 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:23.448769 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.449079 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.459740 11093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } }
I20260812 20:02:23.460013 11093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:23.491472 11140 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:02:23.493464 11140 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:02:23.514874 11140 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Generated new TSK 0
I20260812 20:02:23.518245 11140 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:02:23.644702 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.113s user 0.000s sys 0.038s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:02:23.762923 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.118s user 0.000s sys 0.036s
I20260812 20:02:23.764066 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:24.214437 11161 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:24.214807 11161 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:24.214998 11161 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:24.224280 11161 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:24.224536 11161 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:24.230429 11107 catalog_manager.cc:1295] Loaded cluster ID: 11d9abb606164b81a8f4c8810a2d6849
I20260812 20:02:24.230556 11107 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: loading cluster ID for follower catalog manager: success
I20260812 20:02:24.234629 11107 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: acquiring CA information for follower catalog manager: success
I20260812 20:02:24.237497 11107 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:24.239981 11161 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:24.241595 11161 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:24.243039 11161 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:24.254329 11166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:24.254546 11169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:24.254371 11167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:24.255626 11161 server_base.cc:1061] running on GCE node
I20260812 20:02:24.256333 11161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:24.257488 11161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:24.258734 11161 hybrid_clock.cc:648] HybridClock initialized: now 1786564944258629 us; error 93 us; skew 500 ppm
I20260812 20:02:24.259109 11161 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:24 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564944, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:24.262804 11161 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:24.264740 11161 webserver.cc:533] Webserver started at http://127.25.14.1:45973/ using document root <none> and password file <none>
I20260812 20:02:24.265529 11161 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:24.265695 11161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:24.266047 11161 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:24.268729 11161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:24.269598 11161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:24.275892 11161 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260812 20:02:24.279997 11177 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:24.281739 11161 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:24.281952 11161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:24.282184 11161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:24.283601 11124 catalog_manager.cc:1295] Loaded cluster ID: 11d9abb606164b81a8f4c8810a2d6849
I20260812 20:02:24.283684 11124 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: loading cluster ID for follower catalog manager: success
I20260812 20:02:24.286540 11124 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: acquiring CA information for follower catalog manager: success
I20260812 20:02:24.288472 11124 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:24.304450 11161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:24.309247 11161 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:24.309670 11161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:24.310962 11161 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:24.312844 11161 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:24.312974 11161 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:24.313139 11161 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:24.313215 11161 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:24.359109 11161 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33631
I20260812 20:02:24.359203 11313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33631 every 8 connection(s)
I20260812 20:02:24.360699 11161 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:24.367044 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11161
I20260812 20:02:24.367277 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:02:24 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564944, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:02:24 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564944, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:24 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564944, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:24.393384 11315 heartbeater.cc:344] Connected to a master server at 127.25.14.62:37229
I20260812 20:02:24.393935 11315 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:24.395147 11316 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45939
I20260812 20:02:24.395296 11315 heartbeater.cc:507] Master 127.25.14.62:37229 requested a full tablet report, sending...
I20260812 20:02:24.395370 11316 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:24.395885 11316 heartbeater.cc:507] Master 127.25.14.61:45939 requested a full tablet report, sending...
I20260812 20:02:24.398363 10846 ts_manager.cc:194] Registered new tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:24.399067 10945 ts_manager.cc:194] Registered new tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:24.401957 11314 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35463
I20260812 20:02:24.402114 11314 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:24.402597 11314 heartbeater.cc:507] Master 127.25.14.60:35463 requested a full tablet report, sending...
I20260812 20:02:24.404204 11039 ts_manager.cc:194] Registered new tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:24.406132 11039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:46175
I20260812 20:02:24.415841 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:24.589329 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:38509
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:24.990111 11329 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:24.990548 11329 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:24.990715 11329 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:24.999482 11329 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:24.999658 11329 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:24.999796 11329 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:24.999897 11329 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:25.011982 11329 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:25.013981 11329 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:25.015645 11329 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:25.025215 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:25.025430 11334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:25.025485 11341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:25.026332 11329 server_base.cc:1061] running on GCE node
I20260812 20:02:25.027226 11329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:25.028870 11329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:25.030138 11329 hybrid_clock.cc:648] HybridClock initialized: now 1786564945030083 us; error 37 us; skew 500 ppm
I20260812 20:02:25.030584 11329 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:25.034363 11329 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:02:25.036374 11329 webserver.cc:533] Webserver started at http://127.25.14.59:33401/ using document root <none> and password file <none>
I20260812 20:02:25.037186 11329 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:25.037290 11329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:25.037724 11329 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:25.040501 11329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "20d9416cefcc489090376e3a7834f569"
format_stamp: "Formatted at 2026-08-12 20:02:25 on dist-test-slave-p3t5"
I20260812 20:02:25.041344 11329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "20d9416cefcc489090376e3a7834f569"
format_stamp: "Formatted at 2026-08-12 20:02:25 on dist-test-slave-p3t5"
I20260812 20:02:25.047770 11329 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:02:25.051756 11350 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:25.053469 11329 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 20:02:25.053782 11329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d9416cefcc489090376e3a7834f569"
format_stamp: "Formatted at 2026-08-12 20:02:25 on dist-test-slave-p3t5"
I20260812 20:02:25.054067 11329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:25.077262 11329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:25.081598 11329 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:25.082027 11329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:25.104638 11329 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:38509
I20260812 20:02:25.104663 11405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:38509 every 8 connection(s)
I20260812 20:02:25.106567 11329 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:02:25.111744 11408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:25.111719 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11329
I20260812 20:02:25.120878 11408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:25.143195 11408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:25.156438 11408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:25.170687 11408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:02:25.187942 11408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Bootstrap starting.
I20260812 20:02:25.191520 11408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:25.192884 11408 log.cc:826] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:25.196065 11408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: No bootstrap required, opened a new log
I20260812 20:02:25.201823 11408 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } }
I20260812 20:02:25.202260 11408 raft_consensus.cc:385] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:25.202392 11408 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d9416cefcc489090376e3a7834f569, State: Initialized, Role: FOLLOWER
I20260812 20:02:25.203111 11408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } }
I20260812 20:02:25.204228 11421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } } }
I20260812 20:02:25.204550 11421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:25.205367 11408 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:02:25.216363 11435 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:25.216549 11435 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:25.225409 11329 master_runner.cc:429] Detected that this master 20d9416cefcc489090376e3a7834f569 is joining an existing cluster
I20260812 20:02:25.225581 11329 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:02:25.226574 11329 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:02:25.226992 11329 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:02:25.227207 11329 tablet_replica.cc:333] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: stopping tablet replica
I20260812 20:02:25.228984 11329 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:38509
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:02:25.290302 11039 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:38509: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20d9416cefcc489090376e3a7834f569"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 38509
}
attrs {
promote: true
}
}
dest_uuid: "34a60c59e1f042b9bc12abe767959159"
cas_config_opid_index: -1
I20260812 20:02:25.292135 11039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } }
I20260812 20:02:25.295991 10863 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:02:25.297416 11026 consensus_peers.cc:597] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 -> Peer 20d9416cefcc489090376e3a7834f569 (127.25.14.59:38509): Couldn't send request to peer 20d9416cefcc489090376e3a7834f569. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:02:25.297742 11447 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:02:25.298250 10958 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:02:25.298938 11447 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:02:25.302141 11153 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d9416cefcc489090376e3a7834f569 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.302924 10863 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d9416cefcc489090376e3a7834f569 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.304396 10958 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20d9416cefcc489090376e3a7834f569 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.307500 11153 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:38509
I20260812 20:02:25.308470 11329 master.cc:562] Master@127.25.14.59:38509 shutting down...
I20260812 20:02:25.322646 11454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.323333 11454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:25.326212 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.326748 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:25.328425 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.328751 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:25.329396 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.329452 11455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.329736 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:25.329785 11455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
W20260812 20:02:25.360023 11329 connection.cc:466] server connection from 127.0.0.1:59188 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:59188 (request call id 8) could send its response
I20260812 20:02:25.372680 11329 master.cc:584] Master@127.25.14.59:38509 shutdown complete.
I20260812 20:02:25.372825 11329 master_runner.cc:306] Clearing existing system tablet
I20260812 20:02:25.373293 11329 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:25.373435 11329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:25.376612 11329 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 20:02:25.378469 11458 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:25.378881 11329 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:25.379173 11329 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 20:02:25.379305 11329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d9416cefcc489090376e3a7834f569"
format_stamp: "Formatted at 2026-08-12 20:02:25 on dist-test-slave-p3t5"
I20260812 20:02:25.382759 11329 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:02:25.388044 11329 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:02:25.388286 11329 log.cc:1199] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:02:25.388774 11329 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Deleting consensus metadata
I20260812 20:02:25.389042 11329 master_runner.cc:316] Copying system tablet from 127.25.14.60:35463
I20260812 20:02:25.392010 11329 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.60:35463
I20260812 20:02:25.408943 11314 heartbeater.cc:499] Master 127.25.14.60:35463 was elected leader, sending a full tablet report...
I20260812 20:02:25.409260 11066 tablet_copy_service.cc:140] P 34a60c59e1f042b9bc12abe767959159: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20d9416cefcc489090376e3a7834f569 ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:46546)
I20260812 20:02:25.409574 11066 tablet_copy_service.cc:161] P 34a60c59e1f042b9bc12abe767959159: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20d9416cefcc489090376e3a7834f569 at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:46546: session id = 20d9416cefcc489090376e3a7834f569-00000000000000000000000000000000
I20260812 20:02:25.414180 11066 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:02:25.417012 11329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:02:25.425050 11329 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: tablet copy: Starting download of 0 data blocks...
I20260812 20:02:25.425462 11329 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: tablet copy: Starting download of 1 WAL segments...
I20260812 20:02:25.428607 11329 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:02:25.433873 11066 tablet_copy_service.cc:342] P 34a60c59e1f042b9bc12abe767959159: Request end of tablet copy session 20d9416cefcc489090376e3a7834f569-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:46546
I20260812 20:02:25.434159 11066 tablet_copy_service.cc:434] P 34a60c59e1f042b9bc12abe767959159: ending tablet copy session 20d9416cefcc489090376e3a7834f569-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20d9416cefcc489090376e3a7834f569
W20260812 20:02:25.442123 11329 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:02:25.445730 11329 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:25.448460 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:25.449466 11467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:25.450110 11470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:25.450506 11329 server_base.cc:1061] running on GCE node
I20260812 20:02:25.450894 11329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:25.451584 11329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:25.452796 11329 hybrid_clock.cc:648] HybridClock initialized: now 1786564945452738 us; error 37 us; skew 500 ppm
I20260812 20:02:25.453035 11329 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564945, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:25.455255 11329 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:02:25.458527 11329 webserver.cc:533] Webserver started at http://127.25.14.59:43503/ using document root <none> and password file <none>
I20260812 20:02:25.458976 11329 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:25.459092 11329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:25.462029 11329 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 20:02:25.463850 11481 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:25.464385 11329 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 20:02:25.464602 11329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "20d9416cefcc489090376e3a7834f569"
format_stamp: "Formatted at 2026-08-12 20:02:25 on dist-test-slave-p3t5"
I20260812 20:02:25.464766 11329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:25.471634 11329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:25.474957 11329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:25.496345 11329 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:38509
I20260812 20:02:25.496364 11543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:38509 every 8 connection(s)
I20260812 20:02:25.497360 11329 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:02:25.503139 11544 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:02:25.505527 11544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Bootstrap starting.
I20260812 20:02:25.522912 11544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:02:25.523550 11544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: Bootstrap complete.
I20260812 20:02:25.524838 11544 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } }
I20260812 20:02:25.525159 11544 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20d9416cefcc489090376e3a7834f569, State: Initialized, Role: LEARNER
I20260812 20:02:25.525488 11544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } }
I20260812 20:02:25.527351 11548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.527665 11544 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:25.527750 11548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:02:25.544425 11560 catalog_manager.cc:1295] Loaded cluster ID: 11d9abb606164b81a8f4c8810a2d6849
I20260812 20:02:25.544538 11560 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: loading cluster ID for follower catalog manager: success
I20260812 20:02:25.547451 11560 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: acquiring CA information for follower catalog manager: success
I20260812 20:02:25.550140 11560 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:25.587023 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11161
Aug 12 20:02:25 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:02:25.606948 11308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:25.705976 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11161
I20260812 20:02:25.732583 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33631
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--webserver_port=45973
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:25.811120 11517 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:02:25.812048 11548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a60c59e1f042b9bc12abe767959159. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:25.812366 11548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:02:26.136997 11564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:26.137343 11564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:26.137444 11564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:26.146238 11564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:26.146452 11564 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:26.157742 11564 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45973
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:26.159631 11564 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:26.161247 11564 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:26.172204 11570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:26.172633 11572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:26.173741 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:26.174556 11564 server_base.cc:1061] running on GCE node
I20260812 20:02:26.175300 11564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:26.176455 11564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:26.177752 11564 hybrid_clock.cc:648] HybridClock initialized: now 1786564946177683 us; error 73 us; skew 500 ppm
I20260812 20:02:26.178153 11564 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:26.181908 11564 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:26.183809 11564 webserver.cc:533] Webserver started at http://127.25.14.1:45973/ using document root <none> and password file <none>
I20260812 20:02:26.184732 11564 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:26.184914 11564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:26.191464 11564 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 20:02:26.195433 11587 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:26.197405 11564 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:02:26.197719 11564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:26.198611 11564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:26.239351 11564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:26.243860 11564 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:26.244210 11564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:26.245373 11564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:26.247242 11564 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:26.247386 11564 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:26.247529 11564 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:26.247598 11564 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:26.285786 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: true } } }
I20260812 20:02:26.286197 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:26.291453 11564 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33631
I20260812 20:02:26.291513 11739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33631 every 8 connection(s)
I20260812 20:02:26.293339 11564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:26.297045 11749 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159: attempting to promote NON_VOTER 20d9416cefcc489090376e3a7834f569 to VOTER
I20260812 20:02:26.297173 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11564
I20260812 20:02:26.298349 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } }
I20260812 20:02:26.302444 11517 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 1 LEARNER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:26.303993 11447 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:02:26.323487 10863 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:26.324486 11749 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:02:26.332217 10958 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Refusing update from remote peer 34a60c59e1f042b9bc12abe767959159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:26.333110 11726 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:02:26.342136 11447 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d9416cefcc489090376e3a7834f569 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.348168 11517 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d9416cefcc489090376e3a7834f569 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:26.356009 11740 heartbeater.cc:344] Connected to a master server at 127.25.14.59:38509
I20260812 20:02:26.356511 11740 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:26.356345 10863 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d9416cefcc489090376e3a7834f569 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.357860 11740 heartbeater.cc:507] Master 127.25.14.59:38509 requested a full tablet report, sending...
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:26.362638 11742 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35463
I20260812 20:02:26.362854 11742 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:26.363538 11742 heartbeater.cc:507] Master 127.25.14.60:35463 requested a full tablet report, sending...
I20260812 20:02:26.369421 11766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.369858 11766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:26.372011 11039 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:26.372642 11503 ts_manager.cc:194] Registered new tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:26.372409 10958 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 20d9416cefcc489090376e3a7834f569 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.373251 11039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:32813
I20260812 20:02:26.374963 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.375206 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:26.375859 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.376103 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:26.376668 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.376725 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.376922 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:26.377215 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:26.377589 11748 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45939
I20260812 20:02:26.377822 11748 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:26.378278 11748 heartbeater.cc:507] Master 127.25.14.61:45939 requested a full tablet report, sending...
I20260812 20:02:26.378131 11767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.379416 11767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d9416cefcc489090376e3a7834f569 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:26.379520 11745 heartbeater.cc:344] Connected to a master server at 127.25.14.62:37229
I20260812 20:02:26.379654 11745 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:26.379985 11745 heartbeater.cc:507] Master 127.25.14.62:37229 requested a full tablet report, sending...
I20260812 20:02:26.380487 10946 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:26.380888 11769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:26.381258 11769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d922e9089d44bbebf7129a0ffade3ce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:26.381327 10846 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:26.382773 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:02:26.391914 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:02:26.392109 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10803
I20260812 20:02:26.393121 10888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:26.530821 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10803
I20260812 20:02:26.557322 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:37229
--webserver_interface=127.25.14.62
--webserver_port=35551
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:26.827056 11023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.62:37229: connect: Connection refused (error 111)
W20260812 20:02:26.829034 11023 consensus_peers.cc:597] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 -> Peer 5cdb87ac3fd549dfb13780881f5015b1 (127.25.14.62:37229): Couldn't send request to peer 5cdb87ac3fd549dfb13780881f5015b1. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:37229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:02:26.955294 11771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:26.955709 11771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:26.955870 11771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:26.965039 11771 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:26.965158 11771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:26.965266 11771 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:26.965370 11771 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:26.976898 11771 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35551
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:26.978760 11771 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:26.980340 11771 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:26.990262 11782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:26.990305 11781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:26.991703 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:26.992125 11771 server_base.cc:1061] running on GCE node
I20260812 20:02:26.993106 11771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:26.995146 11771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:26.996402 11771 hybrid_clock.cc:648] HybridClock initialized: now 1786564946996317 us; error 47 us; skew 500 ppm
I20260812 20:02:26.996850 11771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:26 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:27.001152 11771 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:02:27.003695 11771 webserver.cc:533] Webserver started at http://127.25.14.62:35551/ using document root <none> and password file <none>
I20260812 20:02:27.004699 11771 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:27.004889 11771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:27.011996 11771 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 20:02:27.015799 11794 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:27.017818 11771 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:27.018076 11771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5cdb87ac3fd549dfb13780881f5015b1"
format_stamp: "Formatted at 2026-08-12 20:02:21 on dist-test-slave-p3t5"
I20260812 20:02:27.018959 11771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:27.048683 11771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:27.053568 11771 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:27.053954 11771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:27.076295 11771 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:37229
I20260812 20:02:27.076303 11850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:37229 every 8 connection(s)
I20260812 20:02:27.078148 11771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:02:27.081104 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11771
I20260812 20:02:27.081462 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11564
I20260812 20:02:27.087448 11851 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:02:27.091511 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Bootstrap starting.
I20260812 20:02:27.099686 11735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:27.116011 11851 log.cc:826] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:27.130121 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:02:27.130870 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: Bootstrap complete.
I20260812 20:02:27.141981 11851 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } }
I20260812 20:02:27.142804 11851 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb87ac3fd549dfb13780881f5015b1, State: Initialized, Role: FOLLOWER
I20260812 20:02:27.143640 11851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } }
I20260812 20:02:27.144654 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:27.144996 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:27.146045 11851 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:27.157490 11874 catalog_manager.cc:1295] Loaded cluster ID: 11d9abb606164b81a8f4c8810a2d6849
I20260812 20:02:27.157606 11874 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: loading cluster ID for follower catalog manager: success
I20260812 20:02:27.162364 11874 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: acquiring CA information for follower catalog manager: success
I20260812 20:02:27.165341 11874 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:27.221963 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11564
I20260812 20:02:27.249879 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33631
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--webserver_port=45973
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:27.318365 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:27.318694 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:27.330447 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a60c59e1f042b9bc12abe767959159. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:27.330734 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb87ac3fd549dfb13780881f5015b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:02:27.693068 11882 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:27.693549 11882 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:27.693747 11882 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 20:02:27.699121 11883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a60c59e1f042b9bc12abe767959159" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cdb87ac3fd549dfb13780881f5015b1" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 37229 } } peers { permanent_uuid: "2d922e9089d44bbebf7129a0ffade3ce" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 45939 } } peers { permanent_uuid: "34a60c59e1f042b9bc12abe767959159" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 35463 } } peers { permanent_uuid: "20d9416cefcc489090376e3a7834f569" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 38509 } attrs { promote: false } } }
I20260812 20:02:27.699498 11883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a60c59e1f042b9bc12abe767959159 [sys.catalog]: This master's current role is: LEADER
W20260812 20:02:27.703406 11882 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:27.703601 11882 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:27.715443 11882 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45973
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:27.717384 11882 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:27.718961 11882 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:27.730931 11893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:27.730824 11892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:27.730877 11895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:27.732246 11882 server_base.cc:1061] running on GCE node
I20260812 20:02:27.732879 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:27.733940 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:27.735186 11882 hybrid_clock.cc:648] HybridClock initialized: now 1786564947735117 us; error 68 us; skew 500 ppm
I20260812 20:02:27.735540 11882 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:27.739521 11882 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:27.741590 11882 webserver.cc:533] Webserver started at http://127.25.14.1:45973/ using document root <none> and password file <none>
I20260812 20:02:27.742550 11882 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:27.742715 11882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:27.749339 11882 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 20:02:27.752796 11902 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:27.754487 11882 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:27.754662 11882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:27.755424 11882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:27.771098 11882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:27.775496 11882 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:27.775785 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:27.776820 11882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:27.778887 11882 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:27.778992 11882 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:27.779083 11882 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:27.779142 11882 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:27.819444 11882 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33631
I20260812 20:02:27.819542 12043 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33631 every 8 connection(s)
I20260812 20:02:27.821213 11882 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:27.829813 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 11882
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:27 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:27.865540 12048 heartbeater.cc:344] Connected to a master server at 127.25.14.61:45939
I20260812 20:02:27.865638 12046 heartbeater.cc:344] Connected to a master server at 127.25.14.60:35463
I20260812 20:02:27.866048 12046 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:27.867355 12046 heartbeater.cc:507] Master 127.25.14.60:35463 requested a full tablet report, sending...
I20260812 20:02:27.866050 12048 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:27.869001 12048 heartbeater.cc:507] Master 127.25.14.61:45939 requested a full tablet report, sending...
I20260812 20:02:27.870455 11039 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:27.871687 11039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:35803
I20260812 20:02:27.872767 10946 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:27.875349 12047 heartbeater.cc:344] Connected to a master server at 127.25.14.62:37229
I20260812 20:02:27.875512 12047 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:27.875980 12047 heartbeater.cc:507] Master 127.25.14.62:37229 requested a full tablet report, sending...
I20260812 20:02:27.876014 12045 heartbeater.cc:344] Connected to a master server at 127.25.14.59:38509
I20260812 20:02:27.876197 12045 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:27.876600 12045 heartbeater.cc:507] Master 127.25.14.59:38509 requested a full tablet report, sending...
I20260812 20:02:27.878139 11503 ts_manager.cc:194] Re-registered known tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:27.878223 11813 ts_manager.cc:194] Registered new tserver with Master: 7d11428100794b2bba3b12bf00557698 (127.25.14.1:33631)
I20260812 20:02:28.018777 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11882
W20260812 20:02:28.020416 12062 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:28.016142 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.25.14.1:33631)
0812 20:02:28.016863 (+ 721us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:28.016871 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:28.016911 (+ 40us) server_negotiation.cc:408] Connection header received
0812 20:02:28.016984 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:28.016992 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:28.017221 (+ 229us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:28.017412 (+ 191us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:28.018741 (+ 1329us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33304: BlockingRecv error: recv got EOF from 127.0.0.1:33304 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":359,"threads_started":1}
W20260812 20:02:28.021457 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11882: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:33631: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:28.021648 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11882
I20260812 20:02:28.047762 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33631
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--webserver_port=45973
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:28.442983 12064 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:28.443332 12064 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:28.443454 12064 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:02:28.443513 12064 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:28.452710 12064 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:28.452957 12064 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:28.464781 12064 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45973
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:28.466881 12064 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:28.468578 12064 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:28.480944 12074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:28.480850 12071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:28.480792 12072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:28.481904 12064 server_base.cc:1061] running on GCE node
I20260812 20:02:28.482697 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:28.483939 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:28.485258 12064 hybrid_clock.cc:648] HybridClock initialized: now 1786564948485177 us; error 48 us; skew 500 ppm
I20260812 20:02:28.485702 12064 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:28 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:28.489526 12064 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:28.491464 12064 webserver.cc:533] Webserver started at http://127.25.14.1:45973/ using document root <none> and password file <none>
I20260812 20:02:28.492312 12064 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:28.492414 12064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:28.499976 12064 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 20:02:28.503989 12084 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:28.505909 12064 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:28.506106 12064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:28.507113 12064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:28.524963 12064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:28.529836 12064 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:28.530212 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:28.531383 12064 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:28.533244 12064 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:28.533324 12064 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:28.533533 12064 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:28.533597 12064 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:02:28.534102 12064 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:02:28.534197 12064 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:02:28.534291 12064 fault_injection.cc:59] injecting fault for kudu (pid 12064): FLAGS_fault_before_start (process will exit)
I20260812 20:02:28.561056 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:33631
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463
--webserver_port=45973
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:28.953513 12093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:28.953991 12093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:28.954174 12093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:28.963341 12093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:28.963625 12093 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:28.975621 12093 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:33631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=45973
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:37229,127.25.14.61:45939,127.25.14.60:35463,127.25.14.59:38509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:28.977766 12093 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:28.979413 12093 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:28.991317 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:28.991530 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:28.991897 12099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:28.992995 12093 server_base.cc:1061] running on GCE node
I20260812 20:02:28.993854 12093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:28.995131 12093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:28.996424 12093 hybrid_clock.cc:648] HybridClock initialized: now 1786564948996333 us; error 84 us; skew 500 ppm
I20260812 20:02:28.996805 12093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:28 dist-test-slave-p3t5 krb5kdc[10444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:29.000841 12093 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:29.003168 12093 webserver.cc:533] Webserver started at http://127.25.14.1:45973/ using document root <none> and password file <none>
I20260812 20:02:29.004047 12093 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:29.004211 12093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:29.010744 12093 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:02:29.014189 12110 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:29.015779 12093 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:02:29.016002 12093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "7d11428100794b2bba3b12bf00557698"
format_stamp: "Formatted at 2026-08-12 20:02:24 on dist-test-slave-p3t5"
I20260812 20:02:29.016798 12093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:29.045898 12093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:29.050321 12093 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:29.050658 12093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:29.051816 12093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:29.053735 12093 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:29.053884 12093 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:29.054032 12093 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:29.054126 12093 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:29.093575 12093 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33631
I20260812 20:02:29.093701 12241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33631 every 8 connection(s)
I20260812 20:02:29.095429 12093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:29.104652 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12093
I20260812 20:02:29.105021 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12093
Aug 12 20:02:29 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:02:29 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:29 dist-test-slave-p3t5 krb5kdc[10444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
W20260812 20:02:29.141333 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12093: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:33631: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:29.141497 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12093
I20260812 20:02:29.164686 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11771
W20260812 20:02:29.165477 12255 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:29.103546 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:33087 (local address 127.25.14.61:45939)
0812 20:02:29.137919 (+ 34373us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:29.137925 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:29.137946 (+ 21us) server_negotiation.cc:408] Connection header received
0812 20:02:29.138022 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:29.138028 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:29.138127 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:29.138273 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:29.140478 (+ 2205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:02:29.144005 (+ 3527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:02:29.164285 (+ 20280us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:33087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":34234,"thread_start_us":34343,"threads_started":1}
W20260812 20:02:29.165477 12253 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:29.102809 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:58063 (local address 127.25.14.60:35463)
0812 20:02:29.121411 (+ 18602us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:29.121419 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:29.121443 (+ 24us) server_negotiation.cc:408] Connection header received
0812 20:02:29.121516 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:29.121522 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:29.121644 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:29.121786 (+ 142us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:29.130836 (+ 9050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:02:29.132389 (+ 1553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 20:02:29.135806 (+ 3417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 20:02:29.136308 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 20:02:29.164376 (+ 28068us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:58063: BlockingRecv error: recv got EOF from 127.25.14.1:58063 (error 108)
Metrics: {"server-negotiator.queue_time_us":18455,"thread_start_us":19112,"threads_started":1}
W20260812 20:02:29.167342 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11771: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:37229: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:29.167513 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11771
I20260812 20:02:29.195302 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10906
I20260812 20:02:29.196588 10980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:29.312707 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10906
I20260812 20:02:29.342590 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 10999
I20260812 20:02:29.343941 11077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:29.460749 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 10999
W20260812 20:02:29.490383 11485 connection.cc:573] server connection from 127.0.0.1:55740 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:02:29.490629 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 11329
I20260812 20:02:29.491991 11539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:29.614383 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 11329
Shutting down hive metastore.
2026-08-12T20:02:30Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/10 (30873 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/11
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:02:30 dist-test-slave-p3t5 krb5kdc[12268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:02:30 dist-test-slave-p3t5 krb5kdc[12268](info): set up 2 sockets
Aug 12 20:02:30 dist-test-slave-p3t5 krb5kdc[12268](info): commencing operation
krb5kdc: starting...
W20260812 20:02:32.261901 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.186s user 0.003s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:02:32 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:32Z Disabled control of system clock
2026-08-12T20:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:32Z Disabled control of system clock
2026-08-12T20:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:32Z Disabled control of system clock
2026-08-12T20:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:32Z Disabled control of system clock
2026-08-12T20:02:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:02:32Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf
2026-08-12 20:02:40: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskQcgiFK/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Aug 12 20:02:43 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564963, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:02:49 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564969, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 20:02:49 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786564969, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:50.022627 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34641
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:50.412003 12650 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:50.412381 12650 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:50.412472 12650 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:50.421303 12650 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:50.421429 12650 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:50.421486 12650 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:50.421530 12650 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:50.433080 12650 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:50.434957 12650 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:50.436462 12650 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:50.447921 12659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:50.447932 12657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:50.448654 12661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:50.449679 12650 server_base.cc:1061] running on GCE node
I20260812 20:02:50.450645 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:50.452459 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:50.453799 12650 hybrid_clock.cc:648] HybridClock initialized: now 1786564970453647 us; error 114 us; skew 500 ppm
I20260812 20:02:50.454213 12650 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:50 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564970, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:50.457607 12650 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:02:50.459548 12650 webserver.cc:533] Webserver started at http://127.25.14.62:35939/ using document root <none> and password file <none>
I20260812 20:02:50.460393 12650 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:50.460541 12650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:50.460928 12650 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:50.463692 12650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "65ff464f20f443b8a900739cecf4f74e"
format_stamp: "Formatted at 2026-08-12 20:02:50 on dist-test-slave-p3t5"
I20260812 20:02:50.464568 12650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "65ff464f20f443b8a900739cecf4f74e"
format_stamp: "Formatted at 2026-08-12 20:02:50 on dist-test-slave-p3t5"
I20260812 20:02:50.470636 12650 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:02:50.474560 12672 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:50.476023 12650 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:02:50.476212 12650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "65ff464f20f443b8a900739cecf4f74e"
format_stamp: "Formatted at 2026-08-12 20:02:50 on dist-test-slave-p3t5"
I20260812 20:02:50.476450 12650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:50.490751 12650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:50.495235 12650 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:50.495569 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:50.518170 12650 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34641
I20260812 20:02:50.518194 12738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34641 every 8 connection(s)
I20260812 20:02:50.519913 12650 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:02:50.524968 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:50.525971 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12650
I20260812 20:02:50.526191 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:02:50.531631 12739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:50 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564970, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:50.564201 12739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.14.61@KRBTEST.COM; defaulting to no policy
W20260812 20:02:50.574103 12677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
W20260812 20:02:50.576234 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:44181: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:50.625764 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } attempt: 1
W20260812 20:02:50.628160 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:44181: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.61@KRBTEST.COM" created.
I20260812 20:02:50.696470 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } attempt: 2
W20260812 20:02:50.699009 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:44181: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:50.708896 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:44181
--webserver_interface=127.25.14.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:50.790361 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } attempt: 3
W20260812 20:02:50.792907 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:44181: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
I20260812 20:02:50.936223 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } attempt: 4
W20260812 20:02:50.938663 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:44181: Network error: Client connection negotiation failed: client connection to 127.25.14.61:44181: connect: Connection refused (error 111)
W20260812 20:02:51.099133 12749 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:51.099591 12749 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:51.099746 12749 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:51.108717 12749 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:51.108883 12749 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:51.108999 12749 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:51.109077 12749 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:51.120543 12749 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:51.122506 12749 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:51.124037 12749 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:51.133194 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:51.133512 12760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:51.133802 12755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:51.135118 12749 server_base.cc:1061] running on GCE node
I20260812 20:02:51.136042 12749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:51.137867 12749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:51.139194 12749 hybrid_clock.cc:648] HybridClock initialized: now 1786564971139030 us; error 126 us; skew 500 ppm
I20260812 20:02:51.139585 12749 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:51.143005 12749 init.cc:377] Logged in from keytab as kudu/127.25.14.61@KRBTEST.COM (short username kudu)
I20260812 20:02:51.144950 12749 webserver.cc:533] Webserver started at http://127.25.14.61:36623/ using document root <none> and password file <none>
I20260812 20:02:51.145840 12749 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:51.145982 12749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:51.146471 12749 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:51.149251 12749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "59163410375842d0bc31cc221459a891"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.150220 12749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "59163410375842d0bc31cc221459a891"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.156289 12749 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:02:51.160176 12771 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:51.161794 12749 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:02:51.162007 12749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "59163410375842d0bc31cc221459a891"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.162233 12749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:51.192499 12749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:51.196638 12749 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:51.196970 12749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:51.219043 12749 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:44181
I20260812 20:02:51.219082 12833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:44181 every 8 connection(s)
I20260812 20:02:51.220772 12749 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:02:51.221352 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12749
I20260812 20:02:51.221571 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:02:51.227116 12834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:51.236837 12834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:51.237959 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } attempt: 5
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564970, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:51.264206 12739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:51.264742 12834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:02:51.266916 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.60@KRBTEST.COM" created.
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:51.281553 12834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:02:51.283013 12776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
W20260812 20:02:51.287312 12834 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
I20260812 20:02:51.318202 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 1
W20260812 20:02:51.320465 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:51.336728 12834 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 1
W20260812 20:02:51.339182 12834 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.25.14.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.60@KRBTEST.COM" created.
I20260812 20:02:51.388736 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 2
W20260812 20:02:51.391106 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:51.406039 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37297
--webserver_interface=127.25.14.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:51.407454 12834 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 2
W20260812 20:02:51.409485 12834 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
I20260812 20:02:51.497395 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 3
W20260812 20:02:51.499827 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
I20260812 20:02:51.500759 12834 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 3
W20260812 20:02:51.502772 12834 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
I20260812 20:02:51.646003 12834 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 4
W20260812 20:02:51.648558 12834 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
I20260812 20:02:51.677126 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 4
W20260812 20:02:51.680248 12739 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.60:37297: Network error: Client connection negotiation failed: client connection to 127.25.14.60:37297: connect: Connection refused (error 111)
W20260812 20:02:51.803916 12848 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:51.804332 12848 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:51.804497 12848 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:51.813850 12848 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:51.813985 12848 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:51.814076 12848 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:51.814177 12848 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:51.826601 12848 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.60:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.14.60
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:51.828500 12848 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:51.830156 12848 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:51.839964 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:51.839995 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:51.840740 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:51.841919 12848 server_base.cc:1061] running on GCE node
I20260812 20:02:51.842768 12848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:51.844601 12848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:51.846105 12848 hybrid_clock.cc:648] HybridClock initialized: now 1786564971845922 us; error 144 us; skew 500 ppm
I20260812 20:02:51.846598 12848 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:51.850231 12848 init.cc:377] Logged in from keytab as kudu/127.25.14.60@KRBTEST.COM (short username kudu)
I20260812 20:02:51.852073 12848 webserver.cc:533] Webserver started at http://127.25.14.60:37141/ using document root <none> and password file <none>
I20260812 20:02:51.852945 12848 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:51.853135 12848 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:51.853506 12848 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:51.856326 12848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data/instance:
uuid: "6a56d2bf0b044edfa20b7ebdff920859"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.857209 12848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal/instance:
uuid: "6a56d2bf0b044edfa20b7ebdff920859"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.863252 12848 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 20:02:51.867064 12867 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:51.868775 12848 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 20:02:51.869038 12848 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal
uuid: "6a56d2bf0b044edfa20b7ebdff920859"
format_stamp: "Formatted at 2026-08-12 20:02:51 on dist-test-slave-p3t5"
I20260812 20:02:51.869277 12848 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:51.914866 12848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:51.919714 12848 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:51.920083 12848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:51.942873 12848 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.60:37297
I20260812 20:02:51.942880 12930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.60:37297 every 8 connection(s)
I20260812 20:02:51.944661 12848 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/data/info.pb
I20260812 20:02:51.947867 12834 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 5
I20260812 20:02:51.949119 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 12848
I20260812 20:02:51.949357 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-2/wal/instance
I20260812 20:02:51.950441 12931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:51.957511 12739 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } attempt: 5
I20260812 20:02:51.964438 12931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564970, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:51 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:52.010542 12834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: Bootstrap starting.
I20260812 20:02:52.010538 12739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Bootstrap starting.
I20260812 20:02:52.016216 12931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:52.016623 12739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:52.016623 12834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:52.018374 12834 log.cc:826] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:52.019233 12739 log.cc:826] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Log is configured to *not* fsync() on all Append() calls
Aug 12 20:02:52 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:52 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:52.028821 12834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: No bootstrap required, opened a new log
I20260812 20:02:52.037369 12931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:02:52.040707 12739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: No bootstrap required, opened a new log
I20260812 20:02:52.041131 12834 raft_consensus.cc:359] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.041550 12834 raft_consensus.cc:385] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:52.041742 12834 raft_consensus.cc:740] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59163410375842d0bc31cc221459a891, State: Initialized, Role: FOLLOWER
I20260812 20:02:52.042575 12834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
Aug 12 20:02:52 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.60@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:52.053876 12739 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.054401 12739 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:52.054567 12739 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ff464f20f443b8a900739cecf4f74e, State: Initialized, Role: FOLLOWER
I20260812 20:02:52.055862 12945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.056255 12945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.056507 12945 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:02:52.056782 12945 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.057468 12834 sys_catalog.cc:565] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:52.058563 12739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.060827 12945 leader_election.cc:290] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65ff464f20f443b8a900739cecf4f74e (127.25.14.62:34641), 6a56d2bf0b044edfa20b7ebdff920859 (127.25.14.60:37297)
I20260812 20:02:52.063076 12948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.063517 12948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.064234 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: Bootstrap starting.
I20260812 20:02:52.066002 12739 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:52.066636 12709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59163410375842d0bc31cc221459a891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ff464f20f443b8a900739cecf4f74e" is_pre_election: true
I20260812 20:02:52.067313 12709 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59163410375842d0bc31cc221459a891 in term 0.
I20260812 20:02:52.069783 12931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:52.070026 12776 leader_election.cc:304] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59163410375842d0bc31cc221459a891, 65ff464f20f443b8a900739cecf4f74e; no voters:
I20260812 20:02:52.070601 12946 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:02:52.070770 12946 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:02:52.070926 12946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:02:52.071446 12931 log.cc:826] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:52.076932 12946 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.080152 12946 leader_election.cc:290] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [CANDIDATE]: Term 1 election: Requested vote from peers 65ff464f20f443b8a900739cecf4f74e (127.25.14.62:34641), 6a56d2bf0b044edfa20b7ebdff920859 (127.25.14.60:37297)
I20260812 20:02:52.081162 12709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59163410375842d0bc31cc221459a891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ff464f20f443b8a900739cecf4f74e"
I20260812 20:02:52.081455 12709 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:02:52.085335 12709 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59163410375842d0bc31cc221459a891 in term 1.
I20260812 20:02:52.085532 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: No bootstrap required, opened a new log
I20260812 20:02:52.086160 12776 leader_election.cc:304] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59163410375842d0bc31cc221459a891, 65ff464f20f443b8a900739cecf4f74e; no voters:
I20260812 20:02:52.086793 12946 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:02:52.087088 12946 raft_consensus.cc:697] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 1 LEADER]: Becoming Leader. State: Replica: 59163410375842d0bc31cc221459a891, State: Running, Role: LEADER
I20260812 20:02:52.087797 12946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.098495 12945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59163410375842d0bc31cc221459a891. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.098973 12945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:52.102187 12931 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.102679 12931 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:52.102839 12931 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a56d2bf0b044edfa20b7ebdff920859, State: Initialized, Role: FOLLOWER
I20260812 20:02:52.103926 12931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } }
I20260812 20:02:52.111936 12973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.112391 12973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.101931 12749 master_runner.cc:187] Error getting master registration for 127.25.14.62:34641: OK, instance_id { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" instance_seqno: 1786564970511790 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:02:52.113238 12749 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:02:52.115078 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59163410375842d0bc31cc221459a891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56d2bf0b044edfa20b7ebdff920859" is_pre_election: true
I20260812 20:02:52.115049 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59163410375842d0bc31cc221459a891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56d2bf0b044edfa20b7ebdff920859"
I20260812 20:02:52.115957 12898 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59163410375842d0bc31cc221459a891 in term 0.
I20260812 20:02:52.116521 12931 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:02:52.121246 12966 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:52.121459 12966 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:52.121632 12971 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:02:52.124636 12979 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 20:02:52.124933 12971 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:02:52.138527 12650 master_runner.cc:187] Error getting master registration for 127.25.14.60:37297: OK, instance_id { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" instance_seqno: 1786564971936829 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:02:52.148471 12650 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:02:52.155580 12898 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:02:52.158689 12898 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:02:52.159561 12709 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:02:52.159909 12945 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:02:52.160684 12945 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 20:02:52.172515 12874 tablet.cc:2406] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:02:52.174013 12973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59163410375842d0bc31cc221459a891. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.174257 12973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.179399 12982 mvcc.cc:204] Tried to move back new op lower bound from 7317770125937602560 to 7317770125678120960. Current Snapshot: MvccSnapshot[applied={T|T < 7317770125937602560}]
W20260812 20:02:52.180023 12992 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:52.180166 12992 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:52.184201 12973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.184449 12973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.192754 12948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59163410375842d0bc31cc221459a891. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.193066 12948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.194350 12945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.194638 12945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:52.196105 12971 catalog_manager.cc:1383] Generated new cluster ID: b7c6aceee50c4ab39b4fd600b079f952
I20260812 20:02:52.196216 12971 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:02:52.206343 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.206959 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:52.227530 12948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } }
I20260812 20:02:52.227870 12948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:52.269805 12971 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:02:52.271214 12971 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:02:52.286657 12971 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: Generated new TSK 0
I20260812 20:02:52.288070 12971 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:02:52.481094 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:52.965461 13010 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:52.965950 13010 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:52.966094 13010 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:52.976971 13010 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:52.977298 13010 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:52.988603 13010 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:52.990527 13010 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:52.992035 13010 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:53.002838 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:53.002911 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:53.003976 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:53.004488 13010 server_base.cc:1061] running on GCE node
I20260812 20:02:53.005167 13010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:53.006503 13010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:53.007746 13010 hybrid_clock.cc:648] HybridClock initialized: now 1786564973007609 us; error 94 us; skew 500 ppm
I20260812 20:02:53.008077 13010 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:53.011916 13010 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:53.014120 13010 webserver.cc:533] Webserver started at http://127.25.14.1:36775/ using document root <none> and password file <none>
I20260812 20:02:53.014981 13010 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:53.015142 13010 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:53.015550 13010 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:53.018462 13010 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.019381 13010 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.025827 13010 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:02:53.030113 13029 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:53.031924 13010 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 20:02:53.032169 13010 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.032415 13010 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:53.059537 13010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:53.070912 13010 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:53.071274 13010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:53.072378 13010 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:53.074182 13010 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:53.074292 13010 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:53.074457 13010 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:53.074532 13010 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:53.112311 13010 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34227
I20260812 20:02:53.112365 13174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34227 every 8 connection(s)
I20260812 20:02:53.113966 13010 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:53.118618 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13010
I20260812 20:02:53.118870 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 20:02:53.135746 12979 catalog_manager.cc:1295] Loaded cluster ID: b7c6aceee50c4ab39b4fd600b079f952
I20260812 20:02:53.136592 12979 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: loading cluster ID for follower catalog manager: success
I20260812 20:02:53.141052 12979 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: acquiring CA information for follower catalog manager: success
I20260812 20:02:53.145519 12979 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:53.160310 13175 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37297
I20260812 20:02:53.160334 13178 heartbeater.cc:344] Connected to a master server at 127.25.14.61:44181
I20260812 20:02:53.160779 13178 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:53.160776 13175 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:53.162009 13178 heartbeater.cc:507] Master 127.25.14.61:44181 requested a full tablet report, sending...
I20260812 20:02:53.162341 13175 heartbeater.cc:507] Master 127.25.14.60:37297 requested a full tablet report, sending...
I20260812 20:02:53.164736 12792 ts_manager.cc:194] Registered new tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:53.166906 12792 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:33445
I20260812 20:02:53.167405 12886 ts_manager.cc:194] Registered new tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:53.168546 13177 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34641
I20260812 20:02:53.168694 13177 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:53.169286 13177 heartbeater.cc:507] Master 127.25.14.62:34641 requested a full tablet report, sending...
I20260812 20:02:53.171310 12696 ts_manager.cc:194] Registered new tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:53.174386 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:02:53.182832 12992 catalog_manager.cc:1295] Loaded cluster ID: b7c6aceee50c4ab39b4fd600b079f952
I20260812 20:02:53.182960 12992 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: loading cluster ID for follower catalog manager: success
I20260812 20:02:53.187211 12992 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: acquiring CA information for follower catalog manager: success
I20260812 20:02:53.190145 12992 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:02:53.280375 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.59: real 0.104s user 0.006s sys 0.032s
WARNING: no policy specified for HTTP/127.25.14.59@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.59@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.59 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:02:53.384912 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.59: real 0.104s user 0.000s sys 0.037s
I20260812 20:02:53.385887 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:43153
--webserver_interface=127.25.14.59
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:53.788029 13193 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:53.788399 13193 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:53.788486 13193 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:53.797399 13193 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:53.797519 13193 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:53.797576 13193 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:53.797675 13193 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:53.809669 13193 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.59
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.59:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.25.14.59
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:53.811529 13193 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:53.812947 13193 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:53.822259 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:53.822293 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:53.823632 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:53.823975 13193 server_base.cc:1061] running on GCE node
I20260812 20:02:53.824924 13193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:53.826890 13193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:53.828166 13193 hybrid_clock.cc:648] HybridClock initialized: now 1786564973828013 us; error 113 us; skew 500 ppm
I20260812 20:02:53.828575 13193 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:53.832270 13193 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:02:53.834120 13193 webserver.cc:533] Webserver started at http://127.25.14.59:45949/ using document root <none> and password file <none>
I20260812 20:02:53.834898 13193 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:53.835006 13193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:53.835330 13193 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:02:53.838141 13193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/instance:
uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.839035 13193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal/instance:
uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.845245 13193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 20:02:53.849373 13211 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:53.851138 13193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 20:02:53.851365 13193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:53.851657 13193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:53.871601 13193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:53.876636 13193 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:53.876965 13193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:53.899484 13193 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:43153
I20260812 20:02:53.899508 13273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:43153 every 8 connection(s)
I20260812 20:02:53.901186 13193 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:02:53.906816 13274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:02:53.909510 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13193
I20260812 20:02:53.914335 13274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:53.935139 13274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:53.950737 13274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
I20260812 20:02:53.966292 13274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 20:02:53 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:02:53.984242 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Bootstrap starting.
I20260812 20:02:53.987962 13274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Neither blocks nor log segments found. Creating new log.
I20260812 20:02:53.989305 13274 log.cc:826] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:53.992799 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: No bootstrap required, opened a new log
I20260812 20:02:53.998368 13274 raft_consensus.cc:359] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } }
I20260812 20:02:53.998773 13274 raft_consensus.cc:385] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:02:53.998911 13274 raft_consensus.cc:740] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4f6c9688ff4ad8ac772841c41c7cf8, State: Initialized, Role: FOLLOWER
I20260812 20:02:53.999660 13274 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } }
I20260812 20:02:54.000804 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } } }
I20260812 20:02:54.001111 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:54.002151 13274 sys_catalog.cc:565] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:02:54.012135 13300 catalog_manager.cc:1594] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:02:54.012315 13300 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:02:54.021790 13193 master_runner.cc:429] Detected that this master fa4f6c9688ff4ad8ac772841c41c7cf8 is joining an existing cluster
I20260812 20:02:54.021965 13193 catalog_manager.cc:1024] Shutting down the Catalog Manager
I20260812 20:02:54.022914 13193 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:02:54.023373 13193 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:02:54.023620 13193 tablet_replica.cc:333] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: stopping tablet replica
I20260812 20:02:54.025892 13193 master_runner.cc:434] Initiating AddMaster RPC to add 127.25.14.59:43153
Aug 12 20:02:54 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564971, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.61@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
I20260812 20:02:54.100756 12792 catalog_manager.cc:7224] Initiating ChangeConfig request to add master 127.25.14.59:43153: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
member_type: NON_VOTER
last_known_addr {
host: "127.25.14.59"
port: 43153
}
attrs {
promote: true
}
}
dest_uuid: "59163410375842d0bc31cc221459a891"
cas_config_opid_index: -1
I20260812 20:02:54.102684 12792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } }
I20260812 20:02:54.106117 12708 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 20:02:54.107061 13313 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:02:54.108536 12899 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260812 20:02:54.108669 12776 consensus_peers.cc:597] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 -> Peer fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59:43153): Couldn't send request to peer fa4f6c9688ff4ad8ac772841c41c7cf8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:02:54.109938 13313 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 20:02:54.113658 13313 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.114161 12899 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.118237 13313 catalog_manager.cc:7228] Successfully completed master ChangeConfig request to add master 127.25.14.59:43153
I20260812 20:02:54.119305 13193 master.cc:562] Master@127.25.14.59:43153 shutting down...
I20260812 20:02:54.130079 13011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.130425 13011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:54.131312 13011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.131541 13011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:54.134403 12708 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.135367 13318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.135936 13318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:54.138130 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.138586 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:54.139956 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.140443 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:02:54.160192 12266 connection.cc:573] client connection to 127.25.14.59:43153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:02:54.171347 13178 heartbeater.cc:499] Master 127.25.14.61:44181 was elected leader, sending a full tablet report...
I20260812 20:02:54.172570 13193 master.cc:584] Master@127.25.14.59:43153 shutdown complete.
I20260812 20:02:54.172681 13193 master_runner.cc:306] Clearing existing system tablet
I20260812 20:02:54.173290 13193 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:54.173403 13193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:54.177413 13193 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 20:02:54.179577 13323 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:54.180097 13193 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:54.180430 13193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 20:02:54.180539 13193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:54.186563 13193 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:02:54.195822 13193 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:02:54.196125 13193 log.cc:1199] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260812 20:02:54.196779 13193 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Deleting consensus metadata
I20260812 20:02:54.197176 13193 master_runner.cc:316] Copying system tablet from 127.25.14.61:44181
I20260812 20:02:54.201040 13193 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: tablet copy: Beginning tablet copy session from remote peer at address 127.25.14.61:44181
I20260812 20:02:54.217327 12816 tablet_copy_service.cc:140] P 59163410375842d0bc31cc221459a891: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fa4f6c9688ff4ad8ac772841c41c7cf8 ({username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54316)
I20260812 20:02:54.217722 12816 tablet_copy_service.cc:161] P 59163410375842d0bc31cc221459a891: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fa4f6c9688ff4ad8ac772841c41c7cf8 at {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54316: session id = fa4f6c9688ff4ad8ac772841c41c7cf8-00000000000000000000000000000000
I20260812 20:02:54.221328 12816 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 20:02:54.223802 13193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 20:02:54.229559 13193 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: tablet copy: Starting download of 0 data blocks...
I20260812 20:02:54.229966 13193 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: tablet copy: Starting download of 1 WAL segments...
I20260812 20:02:54.232789 13193 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 20:02:54.236445 12816 tablet_copy_service.cc:342] P 59163410375842d0bc31cc221459a891: Request end of tablet copy session fa4f6c9688ff4ad8ac772841c41c7cf8-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.25.14.59@KRBTEST.COM'} at 127.0.0.1:54316
I20260812 20:02:54.236711 12816 tablet_copy_service.cc:434] P 59163410375842d0bc31cc221459a891: ending tablet copy session fa4f6c9688ff4ad8ac772841c41c7cf8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fa4f6c9688ff4ad8ac772841c41c7cf8
W20260812 20:02:54.245055 13193 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260812 20:02:54.249595 13193 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:54.252588 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:54.252647 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:54.254637 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:54.255185 13193 server_base.cc:1061] running on GCE node
I20260812 20:02:54.255522 13193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:54.256325 13193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:54.257526 13193 hybrid_clock.cc:648] HybridClock initialized: now 1786564974257414 us; error 86 us; skew 500 ppm
I20260812 20:02:54.257845 13193 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:54 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.59@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:54.260177 13193 init.cc:377] Logged in from keytab as kudu/127.25.14.59@KRBTEST.COM (short username kudu)
I20260812 20:02:54.262064 13193 webserver.cc:533] Webserver started at http://127.25.14.59:37269/ using document root <none> and password file <none>
I20260812 20:02:54.262434 13193 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:54.262516 13193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:54.265118 13193 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 20:02:54.267148 13344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:54.267671 13193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 20:02:54.267810 13193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:54.268000 13193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:54.284150 13193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:54.287608 13193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:54.308552 13193 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.59:43153
I20260812 20:02:54.308602 13404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.59:43153 every 8 connection(s)
I20260812 20:02:54.309778 13193 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-3/data/info.pb
I20260812 20:02:54.313548 13405 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:02:54.318084 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Bootstrap starting.
I20260812 20:02:54.337297 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:02:54.337918 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: Bootstrap complete.
I20260812 20:02:54.339180 13405 raft_consensus.cc:359] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } }
I20260812 20:02:54.339490 13405 raft_consensus.cc:740] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa4f6c9688ff4ad8ac772841c41c7cf8, State: Initialized, Role: LEARNER
I20260812 20:02:54.339785 13405 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } }
I20260812 20:02:54.340647 13409 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.340968 13409 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: This master's current role is: LEARNER
I20260812 20:02:54.340998 13405 sys_catalog.cc:565] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:54.349843 13422 catalog_manager.cc:1295] Loaded cluster ID: b7c6aceee50c4ab39b4fd600b079f952
I20260812 20:02:54.349962 13422 catalog_manager.cc:1588] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: loading cluster ID for follower catalog manager: success
I20260812 20:02:54.353897 13422 catalog_manager.cc:1610] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: acquiring CA information for follower catalog manager: success
I20260812 20:02:54.356196 13422 catalog_manager.cc:1638] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:54.381858 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13010
Aug 12 20:02:54 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:02:54.400334 13169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:54.519193 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13010
I20260812 20:02:54.552237 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34227
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--webserver_port=36775
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:54.638057 13376 raft_consensus.cc:1217] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260812 20:02:54.638972 13409 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59163410375842d0bc31cc221459a891. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:54.639271 13409 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: This master's current role is: LEARNER
W20260812 20:02:54.956274 13426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:54.956782 13426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:54.956960 13426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:54.966032 13426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:54.966305 13426 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:54.978036 13426 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36775
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:54.979851 13426 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:54.981339 13426 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:54.992740 13439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:54.992645 13434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:54.992681 13433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:54.993975 13426 server_base.cc:1061] running on GCE node
I20260812 20:02:54.994673 13426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:54.996106 13426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:54.997398 13426 hybrid_clock.cc:648] HybridClock initialized: now 1786564974997262 us; error 99 us; skew 500 ppm
I20260812 20:02:54.997817 13426 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:54 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:55.001291 13426 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:55.003351 13426 webserver.cc:533] Webserver started at http://127.25.14.1:36775/ using document root <none> and password file <none>
I20260812 20:02:55.004168 13426 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:55.004331 13426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:55.010494 13426 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:02:55.013541 13449 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:55.015048 13426 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 20:02:55.015254 13426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:55.016119 13426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:55.038797 13426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:55.043690 13426 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:55.044021 13426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:55.045238 13426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:55.047260 13426 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:55.047410 13426 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:55.047559 13426 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:55.047657 13426 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:55.086264 13426 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34227
I20260812 20:02:55.086324 13584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34227 every 8 connection(s)
I20260812 20:02:55.088126 13426 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:55.097985 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13426
Aug 12 20:02:55 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:55 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:55 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
Aug 12 20:02:55 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:02:55.138824 13585 heartbeater.cc:344] Connected to a master server at 127.25.14.59:43153
I20260812 20:02:55.138939 13586 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37297
I20260812 20:02:55.139339 13585 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:55.139335 13586 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:55.140637 13586 heartbeater.cc:507] Master 127.25.14.60:37297 requested a full tablet report, sending...
I20260812 20:02:55.142732 13585 heartbeater.cc:507] Master 127.25.14.59:43153 requested a full tablet report, sending...
I20260812 20:02:55.142982 13588 heartbeater.cc:344] Connected to a master server at 127.25.14.61:44181
I20260812 20:02:55.143139 13588 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:55.143265 12886 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:55.143744 13588 heartbeater.cc:507] Master 127.25.14.61:44181 requested a full tablet report, sending...
I20260812 20:02:55.145107 13366 ts_manager.cc:194] Registered new tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:55.146381 12792 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:55.147390 12792 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:60483
I20260812 20:02:55.148343 13587 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34641
I20260812 20:02:55.148480 13587 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:55.148849 13587 heartbeater.cc:507] Master 127.25.14.62:34641 requested a full tablet report, sending...
I20260812 20:02:55.150072 12696 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:55.157547 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:02:55.167256 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:02:55.167448 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12650
I20260812 20:02:55.168702 12733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:55.190493 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:55.190790 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:55.193850 13603 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891: attempting to promote NON_VOTER fa4f6c9688ff4ad8ac772841c41c7cf8 to VOTER
I20260812 20:02:55.194758 13603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } }
I20260812 20:02:55.197321 12899 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:55.197779 13376 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 1 LEARNER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:55.198051 13602 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:02:55.198551 13316 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 20:02:55.202916 13602 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.203805 13376 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.207247 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.207634 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:55.207973 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.208248 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:55.208268 13602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.208562 13602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:55.209547 12899 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.216300 13609 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.216889 13609 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4f6c9688ff4ad8ac772841c41c7cf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:55.217716 13610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:55.218075 13610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a56d2bf0b044edfa20b7ebdff920859 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:55.291322 12708 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Refusing update from remote peer 59163410375842d0bc31cc221459a891: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 20:02:55.291383 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12650
I20260812 20:02:55.326772 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34641
--webserver_interface=127.25.14.62
--webserver_port=35939
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:55.326985 12776 consensus_peers.cc:597] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 -> Peer 65ff464f20f443b8a900739cecf4f74e (127.25.14.62:34641): Couldn't send request to peer 65ff464f20f443b8a900739cecf4f74e. Status: Network error: Client connection negotiation failed: client connection to 127.25.14.62:34641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 20:02:55.723644 13613 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:55.724067 13613 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:55.724227 13613 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:55.734457 13613 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:02:55.734642 13613 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:55.734774 13613 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:02:55.734869 13613 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:02:55.748070 13613 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=35939
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:55.750303 13613 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:55.752621 13613 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:55.763361 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:55.763827 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:55.764103 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:55.764693 13613 server_base.cc:1061] running on GCE node
I20260812 20:02:55.765578 13613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:55.767571 13613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:55.769063 13613 hybrid_clock.cc:648] HybridClock initialized: now 1786564975768862 us; error 124 us; skew 500 ppm
I20260812 20:02:55.769481 13613 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:55 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564975, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:55.773420 13613 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:02:55.775436 13613 webserver.cc:533] Webserver started at http://127.25.14.62:35939/ using document root <none> and password file <none>
I20260812 20:02:55.776301 13613 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:55.776435 13613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:55.783296 13613 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 20:02:55.786819 13634 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:55.788648 13613 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:02:55.788897 13613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "65ff464f20f443b8a900739cecf4f74e"
format_stamp: "Formatted at 2026-08-12 20:02:50 on dist-test-slave-p3t5"
I20260812 20:02:55.789814 13613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:55.816802 13613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:55.821707 13613 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:55.822109 13613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:55.845414 13613 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34641
I20260812 20:02:55.845439 13697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34641 every 8 connection(s)
I20260812 20:02:55.847335 13613 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:02:55.850482 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13613
I20260812 20:02:55.850888 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13426
I20260812 20:02:55.856496 13698 sys_catalog.cc:263] Verifying existing consensus state
I20260812 20:02:55.860818 13698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Bootstrap starting.
I20260812 20:02:55.869103 13580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:55.881870 13698 log.cc:826] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Log is configured to *not* fsync() on all Append() calls
I20260812 20:02:55.892599 13698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 20:02:55.893239 13698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: Bootstrap complete.
I20260812 20:02:55.900888 13698 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } }
I20260812 20:02:55.901775 13698 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ff464f20f443b8a900739cecf4f74e, State: Initialized, Role: FOLLOWER
I20260812 20:02:55.902742 13698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } }
I20260812 20:02:55.903743 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: NON_VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: true } } }
I20260812 20:02:55.904111 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:55.905050 13698 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:02:55.916621 13720 catalog_manager.cc:1295] Loaded cluster ID: b7c6aceee50c4ab39b4fd600b079f952
I20260812 20:02:55.916743 13720 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: loading cluster ID for follower catalog manager: success
I20260812 20:02:55.921397 13720 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: acquiring CA information for follower catalog manager: success
I20260812 20:02:55.927138 13720 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 20:02:55.987521 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13426
I20260812 20:02:56.016757 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34227
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--webserver_port=36775
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 20:02:56.334622 13668 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fa4f6c9688ff4ad8ac772841c41c7cf8 (127.25.14.59) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:56.345014 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59163410375842d0bc31cc221459a891. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:56.345305 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:02:56.345283 13732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:56.345722 13732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ff464f20f443b8a900739cecf4f74e [sys.catalog]: This master's current role is: FOLLOWER
W20260812 20:02:56.420336 13728 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:56.420698 13728 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:56.420795 13728 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:56.430142 13728 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:56.430421 13728 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:56.441901 13728 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36775
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:56.443763 13728 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:56.445377 13728 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:56.457166 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:56.457221 13737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:56.458257 13728 server_base.cc:1061] running on GCE node
W20260812 20:02:56.458611 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:56.459483 13728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:56.460768 13728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:56.462183 13728 hybrid_clock.cc:648] HybridClock initialized: now 1786564976462025 us; error 119 us; skew 500 ppm
I20260812 20:02:56.462563 13728 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:56.466329 13728 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:56.468487 13728 webserver.cc:533] Webserver started at http://127.25.14.1:36775/ using document root <none> and password file <none>
I20260812 20:02:56.469430 13728 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:56.469596 13728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:56.476203 13728 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 20:02:56.479511 13752 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:56.481141 13728 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 20:02:56.481383 13728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:56.482278 13728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:56.503880 13728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:56.508348 13728 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:56.508728 13728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:56.509904 13728 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:56.511686 13728 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:56.511807 13728 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:56.511945 13728 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:56.512051 13728 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:56.550969 13728 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34227
I20260812 20:02:56.551043 13884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34227 every 8 connection(s)
I20260812 20:02:56.552899 13728 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:56.563743 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13728
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.59@KRBTEST.COM
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.60@KRBTEST.COM
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
Aug 12 20:02:56 dist-test-slave-p3t5 krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564976, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.61@KRBTEST.COM
I20260812 20:02:56.609572 13889 heartbeater.cc:344] Connected to a master server at 127.25.14.61:44181
I20260812 20:02:56.610139 13889 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:56.610508 13887 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34641
I20260812 20:02:56.610693 13887 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:56.611056 13885 heartbeater.cc:344] Connected to a master server at 127.25.14.59:43153
I20260812 20:02:56.611238 13885 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:56.611586 13889 heartbeater.cc:507] Master 127.25.14.61:44181 requested a full tablet report, sending...
I20260812 20:02:56.611588 13887 heartbeater.cc:507] Master 127.25.14.62:34641 requested a full tablet report, sending...
I20260812 20:02:56.611799 13885 heartbeater.cc:507] Master 127.25.14.59:43153 requested a full tablet report, sending...
I20260812 20:02:56.612659 13886 heartbeater.cc:344] Connected to a master server at 127.25.14.60:37297
I20260812 20:02:56.612829 13886 heartbeater.cc:461] Registering TS with master...
I20260812 20:02:56.613327 13886 heartbeater.cc:507] Master 127.25.14.60:37297 requested a full tablet report, sending...
I20260812 20:02:56.614125 12792 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:56.614784 13655 ts_manager.cc:194] Registered new tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:56.615234 12886 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:56.615463 12792 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:49365
I20260812 20:02:56.617036 13366 ts_manager.cc:194] Re-registered known tserver with Master: 3699a2c092024999b90d40ad8d8f58ab (127.25.14.1:34227)
I20260812 20:02:56.726857 13618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59163410375842d0bc31cc221459a891" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "65ff464f20f443b8a900739cecf4f74e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34641 } } peers { permanent_uuid: "59163410375842d0bc31cc221459a891" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 44181 } } peers { permanent_uuid: "6a56d2bf0b044edfa20b7ebdff920859" member_type: VOTER last_known_addr { host: "127.25.14.60" port: 37297 } } peers { permanent_uuid: "fa4f6c9688ff4ad8ac772841c41c7cf8" member_type: VOTER last_known_addr { host: "127.25.14.59" port: 43153 } attrs { promote: false } } }
I20260812 20:02:56.727898 13618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59163410375842d0bc31cc221459a891 [sys.catalog]: This master's current role is: LEADER
I20260812 20:02:56.756832 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13728
W20260812 20:02:56.758281 13914 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:56.754452 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.25.14.1:34227)
0812 20:02:56.755131 (+ 679us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:56.755139 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:56.755172 (+ 33us) server_negotiation.cc:408] Connection header received
0812 20:02:56.755348 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:56.755357 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:56.755599 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:56.755765 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:56.756741 (+ 976us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47022: BlockingRecv error: recv got EOF from 127.0.0.1:47022 (error 108)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":358,"threads_started":1}
W20260812 20:02:56.759308 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13728: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34227: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:56.759464 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13728
I20260812 20:02:56.786165 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34227
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--webserver_port=36775
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:57.189010 13916 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:57.189368 13916 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:57.189465 13916 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260812 20:02:57.189527 13916 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:57.199285 13916 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:57.199523 13916 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:57.211247 13916 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36775
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:57.213045 13916 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:57.214553 13916 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:57.225526 13922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:57.226725 13923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:57.228322 13925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:57.229297 13916 server_base.cc:1061] running on GCE node
I20260812 20:02:57.230059 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:57.231297 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:57.232664 13916 hybrid_clock.cc:648] HybridClock initialized: now 1786564977232494 us; error 119 us; skew 500 ppm
I20260812 20:02:57.233019 13916 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:57 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564977, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:57.236907 13916 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:57.239027 13916 webserver.cc:533] Webserver started at http://127.25.14.1:36775/ using document root <none> and password file <none>
I20260812 20:02:57.239907 13916 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:57.240054 13916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:57.246423 13916 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:02:57.249701 13934 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:57.251415 13916 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:02:57.251626 13916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:57.252416 13916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:57.288690 13916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:57.293119 13916 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:57.293476 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:57.294662 13916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:57.296591 13916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:57.296713 13916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:57.296886 13916 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:57.296968 13916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260812 20:02:57.297432 13916 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 20:02:57.297528 13916 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 20:02:57.297675 13916 fault_injection.cc:59] injecting fault for kudu (pid 13916): FLAGS_fault_before_start (process will exit)
I20260812 20:02:57.320144 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:34227
--local_ip_for_outbound_sockets=127.25.14.1
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297
--webserver_port=36775
--webserver_interface=127.25.14.1
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153 with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:02:57.709857 13943 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:02:57.710261 13943 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:02:57.710436 13943 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:02:57.720041 13943 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:02:57.720315 13943 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:02:57.731808 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39269,127.25.14.21:33583,127.25.14.22:40797,127.25.14.23:46819,127.25.14.24:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:34227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=36775
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34641,127.25.14.61:44181,127.25.14.60:37297,127.25.14.59:43153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:02:57.733647 13943 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:02:57.735186 13943 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:02:57.746647 13951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:57.746613 13950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:02:57.746950 13953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:02:57.747593 13943 server_base.cc:1061] running on GCE node
I20260812 20:02:57.748266 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:02:57.749503 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:02:57.750934 13943 hybrid_clock.cc:648] HybridClock initialized: now 1786564977750758 us; error 126 us; skew 500 ppm
I20260812 20:02:57.751338 13943 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:02:57 dist-test-slave-p3t5 krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564977, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:02:57.755221 13943 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:02:57.757251 13943 webserver.cc:533] Webserver started at http://127.25.14.1:36775/ using document root <none> and password file <none>
I20260812 20:02:57.758131 13943 fs_manager.cc:362] Metadata directory not provided
I20260812 20:02:57.758244 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:02:57.764569 13943 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 20:02:57.767840 13961 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:57.769554 13943 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 20:02:57.769820 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "3699a2c092024999b90d40ad8d8f58ab"
format_stamp: "Formatted at 2026-08-12 20:02:53 on dist-test-slave-p3t5"
I20260812 20:02:57.770622 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:02:57.788764 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:02:57.793404 13943 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:02:57.793814 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:02:57.794996 13943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:02:57.796990 13943 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:02:57.797133 13943 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:57.797272 13943 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:02:57.797380 13943 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:02:57.835884 13943 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:34227
I20260812 20:02:57.835947 14092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:34227 every 8 connection(s)
I20260812 20:02:57.837658 13943 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:02:57.842434 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 13943
I20260812 20:02:57.842795 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13943
W20260812 20:02:57.876927 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13943: Not authorized: Client connection negotiation failed: client connection to 127.25.14.1:34227: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:57.877087 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13943
W20260812 20:02:57.901084 14098 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:57.841500 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:42019 (local address 127.25.14.61:44181)
0812 20:02:57.872395 (+ 30895us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:57.872405 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:57.872441 (+ 36us) server_negotiation.cc:408] Connection header received
0812 20:02:57.873937 (+ 1496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:57.873947 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:57.874084 (+ 137us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:57.874230 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:57.900007 (+ 25777us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:42019: BlockingRecv error: recv got EOF from 127.25.14.1:42019 (error 108)
Metrics: {"server-negotiator.queue_time_us":30712,"thread_start_us":30885,"threads_started":1}
W20260812 20:02:57.901084 14108 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:57.859418 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:48747 (local address 127.25.14.59:43153)
0812 20:02:57.872298 (+ 12880us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:57.872309 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:57.872362 (+ 53us) server_negotiation.cc:408] Connection header received
0812 20:02:57.874516 (+ 2154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:57.874523 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:57.874635 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:57.874773 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:57.900030 (+ 25257us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:48747: BlockingRecv error: recv got EOF from 127.25.14.1:48747 (error 108)
Metrics: {"server-negotiator.queue_time_us":12667,"thread_start_us":13094,"threads_started":1}
W20260812 20:02:57.901078 14099 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:57.841728 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:36587 (local address 127.25.14.62:34641)
0812 20:02:57.872852 (+ 31124us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:57.872857 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:57.872875 (+ 18us) server_negotiation.cc:408] Connection header received
0812 20:02:57.875046 (+ 2171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:57.875054 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:57.875148 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:57.875272 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:57.900049 (+ 24777us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:36587: BlockingRecv error: recv got EOF from 127.25.14.1:36587 (error 108)
Metrics: {"server-negotiator.queue_time_us":31004,"thread_start_us":30906,"threads_started":1}
W20260812 20:02:57.901078 14105 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:57.853953 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.14.1:47123 (local address 127.25.14.60:37297)
0812 20:02:57.854580 (+ 627us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:57.854587 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:57.855498 (+ 911us) server_negotiation.cc:408] Connection header received
0812 20:02:57.874048 (+ 18550us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:57.874058 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:57.874158 (+ 100us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:57.874277 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:57.900034 (+ 25757us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.14.1:47123: BlockingRecv error: recv got EOF from 127.25.14.1:47123 (error 108)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":360,"threads_started":1}
I20260812 20:02:57.902024 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13613
W20260812 20:02:57.904157 25656 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13613: Not authorized: Client connection negotiation failed: client connection to 127.25.14.62:34641: server requires authentication, but client does not have Kerberos credentials available
I20260812 20:02:57.904301 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13613
W20260812 20:02:57.904313 14099 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 20:02:57.902904 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.25.14.62:34641)
0812 20:02:57.903100 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0812 20:02:57.903105 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0812 20:02:57.903124 (+ 19us) server_negotiation.cc:408] Connection header received
0812 20:02:57.903249 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 20:02:57.903254 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 20:02:57.903323 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 20:02:57.903407 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0812 20:02:57.904083 (+ 676us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 20:02:57.931967 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12749
I20260812 20:02:57.933297 12828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:58.054600 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12749
W20260812 20:02:58.084455 12870 connection.cc:573] server connection from 127.0.0.1:37240 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 20:02:58.084929 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 12848
I20260812 20:02:58.086265 12925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:58.228163 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 12848
I20260812 20:02:58.256791 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 13193
I20260812 20:02:58.258260 13399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:02:58.376531 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 13193
Shutting down hive metastore.
2026-08-12T20:02:58Z chronyd exiting
2026-08-12T20:02:58Z chronyd exiting
2026-08-12T20:02:58Z chronyd exiting
2026-08-12T20:02:58Z chronyd exiting
2026-08-12T20:02:58Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/11 (28800 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:02:58 dist-test-slave-p3t5 krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:02:58 dist-test-slave-p3t5 krb5kdc[14121](info): set up 2 sockets
Aug 12 20:02:58 dist-test-slave-p3t5 krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260812 20:03:01.126997 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.250s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:01 dist-test-slave-p3t5 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:01.626187 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38517
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38517
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:02.021154 14138 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:02.021544 14138 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:02.021773 14138 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:02.031905 14138 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:02.032045 14138 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:02.032172 14138 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:02.032248 14138 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:02.045298 14138 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38517
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:02.047217 14138 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:02.048763 14138 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:02.059689 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:02.059749 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:02.059775 14149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:02.061182 14138 server_base.cc:1061] running on GCE node
I20260812 20:03:02.062217 14138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:02.063799 14138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:02.065053 14138 hybrid_clock.cc:648] HybridClock initialized: now 1786564982064995 us; error 46 us; skew 500 ppm
I20260812 20:03:02.065428 14138 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:02 dist-test-slave-p3t5 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:02.069067 14138 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:02.070963 14138 webserver.cc:533] Webserver started at http://127.25.14.62:43311/ using document root <none> and password file <none>
I20260812 20:03:02.071816 14138 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:02.071985 14138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:02.072391 14138 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:02.075371 14138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "ff4bf9e8d798494ea4dfcb11af3515f2"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.076288 14138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "ff4bf9e8d798494ea4dfcb11af3515f2"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.082733 14138 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 20:03:02.086884 14158 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:02.088725 14138 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 20:03:02.089001 14138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "ff4bf9e8d798494ea4dfcb11af3515f2"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.089248 14138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:02.114526 14138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:02.119055 14138 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:02.119390 14138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:02.142668 14138 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38517
I20260812 20:03:02.142697 14221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38517 every 8 connection(s)
I20260812 20:03:02.144536 14138 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:02.148422 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14138
I20260812 20:03:02.148710 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:02.150453 14222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:02.163666 14222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2: Bootstrap starting.
Aug 12 20:03:02 dist-test-slave-p3t5 krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:02.168033 14222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:02.169335 14222 log.cc:826] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:02.173244 14222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2: No bootstrap required, opened a new log
I20260812 20:03:02.179888 14222 raft_consensus.cc:359] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } }
I20260812 20:03:02.180338 14222 raft_consensus.cc:385] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:02.180476 14222 raft_consensus.cc:740] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4bf9e8d798494ea4dfcb11af3515f2, State: Initialized, Role: FOLLOWER
I20260812 20:03:02.181310 14222 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } }
I20260812 20:03:02.181587 14222 raft_consensus.cc:399] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:02.181769 14222 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:02.181993 14222 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:02.184517 14222 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } }
I20260812 20:03:02.185151 14222 leader_election.cc:304] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4bf9e8d798494ea4dfcb11af3515f2; no voters:
I20260812 20:03:02.186264 14222 leader_election.cc:290] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:02.186414 14228 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:02.187239 14228 raft_consensus.cc:697] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [term 1 LEADER]: Becoming Leader. State: Replica: ff4bf9e8d798494ea4dfcb11af3515f2, State: Running, Role: LEADER
I20260812 20:03:02.187904 14228 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } }
I20260812 20:03:02.189116 14222 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:02.191816 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4bf9e8d798494ea4dfcb11af3515f2. Latest consensus state: current_term: 1 leader_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } } }
I20260812 20:03:02.191785 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4bf9e8d798494ea4dfcb11af3515f2" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38517 } } }
I20260812 20:03:02.192257 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:02.192247 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:02.203333 14234 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:02.208753 14234 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:02.219897 14234 catalog_manager.cc:1383] Generated new cluster ID: 4ba81c8b39bc4ce2abdb85301438dd72
I20260812 20:03:02.220047 14234 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:02.238600 14234 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:02.240026 14234 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:02.255977 14234 catalog_manager.cc:6092] T 00000000000000000000000000000000 P ff4bf9e8d798494ea4dfcb11af3515f2: Generated new TSK 0
I20260812 20:03:02.257099 14234 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:02.444540 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38517
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:02.847199 14253 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:02.847622 14253 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:02.847800 14253 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:02.856688 14253 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:02.857009 14253 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:02.868505 14253 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:38517
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:02.870493 14253 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:02.872018 14253 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:02.883203 14262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:02.883487 14260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:02.883271 14259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:02.885221 14253 server_base.cc:1061] running on GCE node
I20260812 20:03:02.885951 14253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:02.887156 14253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:02.888415 14253 hybrid_clock.cc:648] HybridClock initialized: now 1786564982888369 us; error 46 us; skew 500 ppm
I20260812 20:03:02.888784 14253 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:02 dist-test-slave-p3t5 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:02.892517 14253 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:02.894570 14253 webserver.cc:533] Webserver started at http://127.25.14.1:44683/ using document root <none> and password file <none>
I20260812 20:03:02.895545 14253 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:02.895692 14253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:02.896087 14253 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:02.898805 14253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "d157fdab67444aa98b979e51c1212b5b"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.899675 14253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "d157fdab67444aa98b979e51c1212b5b"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.905716 14253 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:03:02.909569 14271 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:02.911186 14253 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:02.911412 14253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "d157fdab67444aa98b979e51c1212b5b"
format_stamp: "Formatted at 2026-08-12 20:03:02 on dist-test-slave-p3t5"
I20260812 20:03:02.911646 14253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:02.933404 14253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:02.937785 14253 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:02.938126 14253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:02.939193 14253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:02.941037 14253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:02.941154 14253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:02.941320 14253 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:02.941406 14253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:02.980348 14253 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:41005
I20260812 20:03:02.980387 14406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:41005 every 8 connection(s)
I20260812 20:03:02.981976 14253 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:02.992014 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14253
I20260812 20:03:02.992304 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:02 dist-test-slave-p3t5 krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564982, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:03.005019 14407 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38517
I20260812 20:03:03.005452 14407 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:03.006486 14407 heartbeater.cc:507] Master 127.25.14.62:38517 requested a full tablet report, sending...
I20260812 20:03:03.009044 14179 ts_manager.cc:194] Registered new tserver with Master: d157fdab67444aa98b979e51c1212b5b (127.25.14.1:41005)
I20260812 20:03:03.010238 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 20:03:03.010923 14179 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:41823
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:03.094249 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14253
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:03.117525 14402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:03.204880 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14253
I20260812 20:03:03.229892 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14138
I20260812 20:03:03.231392 14214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:03.340754 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14138
2026-08-12T20:03:03Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0 (4515 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14424](info): set up 2 sockets
Aug 12 20:03:03 dist-test-slave-p3t5 krb5kdc[14424](info): commencing operation
krb5kdc: starting...
W20260812 20:03:05.571890 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.180s user 0.000s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:05 dist-test-slave-p3t5 krb5kdc[14424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:06.074332 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35021
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:35021
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:06.463940 14443 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:06.464332 14443 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:06.464507 14443 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:06.473381 14443 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:06.473523 14443 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:06.473632 14443 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:06.473765 14443 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:06.485224 14443 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:35021
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:06.487192 14443 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:06.488804 14443 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:06.499125 14449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:06.499189 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:06.499328 14450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:06.500259 14443 server_base.cc:1061] running on GCE node
I20260812 20:03:06.501235 14443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:06.503013 14443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:06.504289 14443 hybrid_clock.cc:648] HybridClock initialized: now 1786564986504230 us; error 81 us; skew 500 ppm
I20260812 20:03:06.504706 14443 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:06 dist-test-slave-p3t5 krb5kdc[14424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564986, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:06.508466 14443 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:06.510416 14443 webserver.cc:533] Webserver started at http://127.25.14.62:33689/ using document root <none> and password file <none>
I20260812 20:03:06.511296 14443 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:06.511451 14443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:06.511876 14443 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:06.515178 14443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "c6348bfa1b2d48129c4df64e6a089be0"
format_stamp: "Formatted at 2026-08-12 20:03:06 on dist-test-slave-p3t5"
I20260812 20:03:06.516148 14443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "c6348bfa1b2d48129c4df64e6a089be0"
format_stamp: "Formatted at 2026-08-12 20:03:06 on dist-test-slave-p3t5"
I20260812 20:03:06.522439 14443 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:03:06.526772 14459 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:06.528599 14443 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:03:06.528856 14443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "c6348bfa1b2d48129c4df64e6a089be0"
format_stamp: "Formatted at 2026-08-12 20:03:06 on dist-test-slave-p3t5"
I20260812 20:03:06.529116 14443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:06.553781 14443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:06.558645 14443 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:06.559046 14443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:06.585693 14443 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:35021
I20260812 20:03:06.585728 14518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:35021 every 8 connection(s)
I20260812 20:03:06.587581 14443 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:06.593164 14519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:06.597880 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14443
I20260812 20:03:06.598115 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:06.607623 14519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0: Bootstrap starting.
I20260812 20:03:06.612710 14519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:06.614387 14519 log.cc:826] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0: Log is configured to *not* fsync() on all Append() calls
Aug 12 20:03:06 dist-test-slave-p3t5 krb5kdc[14424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:06.617688 14519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0: No bootstrap required, opened a new log
I20260812 20:03:06.623628 14519 raft_consensus.cc:359] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } }
I20260812 20:03:06.624096 14519 raft_consensus.cc:385] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:06.624236 14519 raft_consensus.cc:740] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6348bfa1b2d48129c4df64e6a089be0, State: Initialized, Role: FOLLOWER
I20260812 20:03:06.625177 14519 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } }
I20260812 20:03:06.625492 14519 raft_consensus.cc:399] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:06.625665 14519 raft_consensus.cc:493] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:06.625869 14519 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:06.628594 14519 raft_consensus.cc:515] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } }
I20260812 20:03:06.629172 14519 leader_election.cc:304] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6348bfa1b2d48129c4df64e6a089be0; no voters:
I20260812 20:03:06.630000 14519 leader_election.cc:290] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:06.630107 14526 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:06.631024 14526 raft_consensus.cc:697] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [term 1 LEADER]: Becoming Leader. State: Replica: c6348bfa1b2d48129c4df64e6a089be0, State: Running, Role: LEADER
I20260812 20:03:06.631699 14526 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } }
I20260812 20:03:06.632454 14519 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:06.635241 14527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6348bfa1b2d48129c4df64e6a089be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } } }
I20260812 20:03:06.635569 14527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:06.636339 14528 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6348bfa1b2d48129c4df64e6a089be0. Latest consensus state: current_term: 1 leader_uuid: "c6348bfa1b2d48129c4df64e6a089be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6348bfa1b2d48129c4df64e6a089be0" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 35021 } } }
I20260812 20:03:06.636739 14528 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:06.639832 14536 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:06.647634 14536 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:06.660087 14536 catalog_manager.cc:1383] Generated new cluster ID: 37d554ef7ddb44b88628fcea387f09c8
I20260812 20:03:06.660245 14536 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:06.670033 14536 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:06.671901 14536 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:06.686805 14536 catalog_manager.cc:6092] T 00000000000000000000000000000000 P c6348bfa1b2d48129c4df64e6a089be0: Generated new TSK 0
I20260812 20:03:06.688104 14536 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:06.877692 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:35021
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:07.264510 14557 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:07.264919 14557 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:07.265098 14557 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:07.273764 14557 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:07.274041 14557 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:07.284991 14557 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:07.286857 14557 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:07.288384 14557 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:07.299247 14565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:07.299392 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:07.299211 14564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:07.300848 14557 server_base.cc:1061] running on GCE node
I20260812 20:03:07.301513 14557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:07.302794 14557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:07.304044 14557 hybrid_clock.cc:648] HybridClock initialized: now 1786564987303973 us; error 55 us; skew 500 ppm
I20260812 20:03:07.304401 14557 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:07.308111 14557 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:07.310164 14557 webserver.cc:533] Webserver started at http://127.25.14.1:38351/ using document root <none> and password file <none>
I20260812 20:03:07.311039 14557 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:07.311156 14557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:07.311611 14557 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:07.314436 14557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "61715037af2b42a68a671a82d4e6f7a9"
format_stamp: "Formatted at 2026-08-12 20:03:07 on dist-test-slave-p3t5"
I20260812 20:03:07.315331 14557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "61715037af2b42a68a671a82d4e6f7a9"
format_stamp: "Formatted at 2026-08-12 20:03:07 on dist-test-slave-p3t5"
I20260812 20:03:07.321400 14557 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:07.325250 14580 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:07.327031 14557 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:07.327252 14557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "61715037af2b42a68a671a82d4e6f7a9"
format_stamp: "Formatted at 2026-08-12 20:03:07 on dist-test-slave-p3t5"
I20260812 20:03:07.327490 14557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:07.345415 14557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:07.349761 14557 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:07.350103 14557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:07.351269 14557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:07.352975 14557 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:07.353091 14557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:07.353251 14557 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:07.353327 14557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:07.390810 14557 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37619
I20260812 20:03:07.390863 14718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37619 every 8 connection(s)
I20260812 20:03:07.392473 14557 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:07.401939 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14557
I20260812 20:03:07.402184 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:07.415073 14719 heartbeater.cc:344] Connected to a master server at 127.25.14.62:35021
I20260812 20:03:07.415512 14719 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:07.416622 14719 heartbeater.cc:507] Master 127.25.14.62:35021 requested a full tablet report, sending...
I20260812 20:03:07.419344 14477 ts_manager.cc:194] Registered new tserver with Master: 61715037af2b42a68a671a82d4e6f7a9 (127.25.14.1:37619)
I20260812 20:03:07.420985 14477 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:55039
I20260812 20:03:07.431485 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:07.515851 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14557
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:07.534654 14712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:07.622319 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14557
I20260812 20:03:07.647840 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14443
I20260812 20:03:07.649200 14514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:07.759488 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14443
2026-08-12T20:03:07Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1 (4421 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14733](info): set up 2 sockets
Aug 12 20:03:07 dist-test-slave-p3t5 krb5kdc[14733](info): commencing operation
krb5kdc: starting...
W20260812 20:03:10.012670 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.199s user 0.003s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:10 dist-test-slave-p3t5 krb5kdc[14733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:10.518179 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34349
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:34349
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:10.922839 14753 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:10.923216 14753 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:10.923430 14753 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:10.933168 14753 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:10.933290 14753 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:10.933396 14753 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:10.933470 14753 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:10.945137 14753 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:34349
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:10.947079 14753 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:10.948616 14753 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:10.957329 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:10.957381 14760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:10.958092 14763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:10.959190 14753 server_base.cc:1061] running on GCE node
I20260812 20:03:10.960193 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:10.961858 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:10.963213 14753 hybrid_clock.cc:648] HybridClock initialized: now 1786564990963130 us; error 80 us; skew 500 ppm
I20260812 20:03:10.963800 14753 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:10 dist-test-slave-p3t5 krb5kdc[14733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564990, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:10.967414 14753 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:10.969267 14753 webserver.cc:533] Webserver started at http://127.25.14.62:36121/ using document root <none> and password file <none>
I20260812 20:03:10.970211 14753 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:10.970369 14753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:10.970793 14753 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:10.973565 14753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "77978cde50354b85bd189421cb831e45"
format_stamp: "Formatted at 2026-08-12 20:03:10 on dist-test-slave-p3t5"
I20260812 20:03:10.974571 14753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "77978cde50354b85bd189421cb831e45"
format_stamp: "Formatted at 2026-08-12 20:03:10 on dist-test-slave-p3t5"
I20260812 20:03:10.980930 14753 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:10.985204 14770 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:10.986950 14753 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:10.987193 14753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "77978cde50354b85bd189421cb831e45"
format_stamp: "Formatted at 2026-08-12 20:03:10 on dist-test-slave-p3t5"
I20260812 20:03:10.987437 14753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:11.001909 14753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:11.006434 14753 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:11.006803 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:11.030517 14753 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:34349
I20260812 20:03:11.030544 14830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:34349 every 8 connection(s)
I20260812 20:03:11.032295 14753 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:11.037542 14831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:11.042016 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14753
I20260812 20:03:11.042280 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:11.051075 14831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45: Bootstrap starting.
I20260812 20:03:11.055111 14831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:11.056630 14831 log.cc:826] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45: Log is configured to *not* fsync() on all Append() calls
Aug 12 20:03:11 dist-test-slave-p3t5 krb5kdc[14733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:11.060248 14831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45: No bootstrap required, opened a new log
I20260812 20:03:11.066570 14831 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } }
I20260812 20:03:11.067062 14831 raft_consensus.cc:385] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:11.067209 14831 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77978cde50354b85bd189421cb831e45, State: Initialized, Role: FOLLOWER
I20260812 20:03:11.068042 14831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } }
I20260812 20:03:11.068359 14831 raft_consensus.cc:399] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:11.068533 14831 raft_consensus.cc:493] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:11.068739 14831 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:11.071309 14831 raft_consensus.cc:515] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } }
I20260812 20:03:11.071895 14831 leader_election.cc:304] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77978cde50354b85bd189421cb831e45; no voters:
I20260812 20:03:11.072602 14831 leader_election.cc:290] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:11.072770 14838 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:11.073661 14838 raft_consensus.cc:697] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [term 1 LEADER]: Becoming Leader. State: Replica: 77978cde50354b85bd189421cb831e45, State: Running, Role: LEADER
I20260812 20:03:11.074445 14838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } }
I20260812 20:03:11.075125 14831 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:11.078028 14839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77978cde50354b85bd189421cb831e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } } }
I20260812 20:03:11.078384 14839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:11.078819 14840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77978cde50354b85bd189421cb831e45. Latest consensus state: current_term: 1 leader_uuid: "77978cde50354b85bd189421cb831e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77978cde50354b85bd189421cb831e45" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 34349 } } }
I20260812 20:03:11.079138 14840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:11.081477 14844 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:11.089243 14844 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:11.100665 14844 catalog_manager.cc:1383] Generated new cluster ID: 62cfc7d4a9b54c4f929b691475b0e95e
I20260812 20:03:11.100829 14844 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:11.127115 14844 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:11.128620 14844 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:11.142261 14844 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 77978cde50354b85bd189421cb831e45: Generated new TSK 0
I20260812 20:03:11.143267 14844 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:03:11.256227 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.104s user 0.000s sys 0.035s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:03:11.360535 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.104s user 0.000s sys 0.035s
I20260812 20:03:11.361490 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:34349
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:11.757067 14865 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:11.757571 14865 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:11.757757 14865 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:11.766551 14865 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:11.766777 14865 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:11.778070 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:34349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:11.780246 14865 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:11.782434 14865 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:11.794359 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:11.794317 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:11.795282 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:11.795715 14865 server_base.cc:1061] running on GCE node
I20260812 20:03:11.796439 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:11.797776 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:11.799090 14865 hybrid_clock.cc:648] HybridClock initialized: now 1786564991799034 us; error 35 us; skew 500 ppm
I20260812 20:03:11.799459 14865 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:11 dist-test-slave-p3t5 krb5kdc[14733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564991, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:11.803531 14865 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:11.805748 14865 webserver.cc:533] Webserver started at http://127.25.14.1:33163/ using document root <none> and password file <none>
I20260812 20:03:11.806627 14865 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:11.806751 14865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:11.807093 14865 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:11.809894 14865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "17941f7c0d214e1eaa5b4d9afb7decaa"
format_stamp: "Formatted at 2026-08-12 20:03:11 on dist-test-slave-p3t5"
I20260812 20:03:11.810753 14865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "17941f7c0d214e1eaa5b4d9afb7decaa"
format_stamp: "Formatted at 2026-08-12 20:03:11 on dist-test-slave-p3t5"
I20260812 20:03:11.816835 14865 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:11.820644 14883 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:11.822238 14865 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:11.822405 14865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "17941f7c0d214e1eaa5b4d9afb7decaa"
format_stamp: "Formatted at 2026-08-12 20:03:11 on dist-test-slave-p3t5"
I20260812 20:03:11.822595 14865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:11.840638 14865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:11.845187 14865 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:11.845467 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:11.846534 14865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:11.848264 14865 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:11.848348 14865 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:11.848439 14865 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:11.848485 14865 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:11.886315 14865 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:32833
I20260812 20:03:11.886408 15014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:32833 every 8 connection(s)
I20260812 20:03:11.887923 14865 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:11.895020 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 14865
I20260812 20:03:11.895237 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:11 dist-test-slave-p3t5 krb5kdc[14733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564991, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:11.910578 15015 heartbeater.cc:344] Connected to a master server at 127.25.14.62:34349
I20260812 20:03:11.911059 15015 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:11.912106 15015 heartbeater.cc:507] Master 127.25.14.62:34349 requested a full tablet report, sending...
I20260812 20:03:11.914520 14787 ts_manager.cc:194] Registered new tserver with Master: 17941f7c0d214e1eaa5b4d9afb7decaa (127.25.14.1:32833)
I20260812 20:03:11.916150 14787 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:45477
I20260812 20:03:11.924597 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:11 dist-test-slave-p3t5 krb5kdc[14733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:12 dist-test-slave-p3t5 krb5kdc[14733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:12.007972 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14865
Aug 12 20:03:12 dist-test-slave-p3t5 krb5kdc[14733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:12.027745 15009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:12.122001 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14865
I20260812 20:03:12.146684 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 14753
I20260812 20:03:12.148094 14822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:12.256580 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 14753
2026-08-12T20:03:12Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2 (4495 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:12 dist-test-slave-p3t5 krb5kdc[15032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:12 dist-test-slave-p3t5 krb5kdc[15032](info): set up 2 sockets
Aug 12 20:03:12 dist-test-slave-p3t5 krb5kdc[15032](info): commencing operation
krb5kdc: starting...
W20260812 20:03:14.499766 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.190s user 0.000s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:14 dist-test-slave-p3t5 krb5kdc[15032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:15.015019 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40861
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40861
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:15.409509 15051 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:15.410045 15051 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:15.410214 15051 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:15.419394 15051 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:15.419569 15051 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:15.419684 15051 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:15.419771 15051 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:15.431402 15051 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40861
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:15.433336 15051 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:15.434957 15051 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:15.444903 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:15.444867 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:15.444824 15058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:15.446038 15051 server_base.cc:1061] running on GCE node
I20260812 20:03:15.446969 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:15.448968 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:15.450217 15051 hybrid_clock.cc:648] HybridClock initialized: now 1786564995450152 us; error 59 us; skew 500 ppm
I20260812 20:03:15.450671 15051 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:15 dist-test-slave-p3t5 krb5kdc[15032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:15.454337 15051 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:15.456252 15051 webserver.cc:533] Webserver started at http://127.25.14.62:44657/ using document root <none> and password file <none>
I20260812 20:03:15.457073 15051 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:15.457208 15051 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:15.457736 15051 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:15.460453 15051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "3acb231da7184a95980a3d9311827c32"
format_stamp: "Formatted at 2026-08-12 20:03:15 on dist-test-slave-p3t5"
I20260812 20:03:15.461290 15051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "3acb231da7184a95980a3d9311827c32"
format_stamp: "Formatted at 2026-08-12 20:03:15 on dist-test-slave-p3t5"
I20260812 20:03:15.467518 15051 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 20:03:15.471748 15069 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:15.473534 15051 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 20:03:15.473841 15051 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3acb231da7184a95980a3d9311827c32"
format_stamp: "Formatted at 2026-08-12 20:03:15 on dist-test-slave-p3t5"
I20260812 20:03:15.474081 15051 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:15.502992 15051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:15.511569 15051 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:15.512042 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:15.535274 15051 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40861
I20260812 20:03:15.535308 15125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40861 every 8 connection(s)
I20260812 20:03:15.536995 15051 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:15.539549 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15051
I20260812 20:03:15.539781 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:15.542414 15126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:15.553519 15126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32: Bootstrap starting.
Aug 12 20:03:15 dist-test-slave-p3t5 krb5kdc[15032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:15.558090 15126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:15.559536 15126 log.cc:826] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:15.563486 15126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32: No bootstrap required, opened a new log
I20260812 20:03:15.570058 15126 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } }
I20260812 20:03:15.570438 15126 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:15.570536 15126 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acb231da7184a95980a3d9311827c32, State: Initialized, Role: FOLLOWER
I20260812 20:03:15.571223 15126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } }
I20260812 20:03:15.571430 15126 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:15.571540 15126 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:15.571672 15126 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:15.574330 15126 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } }
I20260812 20:03:15.574918 15126 leader_election.cc:304] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3acb231da7184a95980a3d9311827c32; no voters:
I20260812 20:03:15.575654 15126 leader_election.cc:290] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:15.575811 15131 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:15.576689 15131 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [term 1 LEADER]: Becoming Leader. State: Replica: 3acb231da7184a95980a3d9311827c32, State: Running, Role: LEADER
I20260812 20:03:15.577306 15131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } }
I20260812 20:03:15.578306 15126 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:15.580598 15133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3acb231da7184a95980a3d9311827c32. Latest consensus state: current_term: 1 leader_uuid: "3acb231da7184a95980a3d9311827c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } } }
I20260812 20:03:15.581391 15133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:15.581602 15132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3acb231da7184a95980a3d9311827c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acb231da7184a95980a3d9311827c32" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40861 } } }
I20260812 20:03:15.582425 15132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:15.587096 15139 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:15.593240 15139 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:15.604179 15139 catalog_manager.cc:1383] Generated new cluster ID: 6c7c4155e4d0470996fd5ae53eb3b35f
I20260812 20:03:15.604358 15139 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:15.625945 15139 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:15.627422 15139 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:15.638239 15139 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 3acb231da7184a95980a3d9311827c32: Generated new TSK 0
I20260812 20:03:15.639238 15139 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:15.829764 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40861
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:16.223065 15159 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:16.223419 15159 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:16.223515 15159 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:16.232379 15159 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:16.232599 15159 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:16.243952 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:40861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:16.245764 15159 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:16.247262 15159 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:16.258306 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:16.258348 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:16.258780 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:16.259979 15159 server_base.cc:1061] running on GCE node
I20260812 20:03:16.260622 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:16.261927 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:16.263147 15159 hybrid_clock.cc:648] HybridClock initialized: now 1786564996263070 us; error 64 us; skew 500 ppm
I20260812 20:03:16.263461 15159 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:16.267064 15159 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:16.268894 15159 webserver.cc:533] Webserver started at http://127.25.14.1:45277/ using document root <none> and password file <none>
I20260812 20:03:16.269819 15159 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:16.269960 15159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:16.270428 15159 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:16.273254 15159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "2f24c96fdace4b1b88eebe2c7a1c2ba4"
format_stamp: "Formatted at 2026-08-12 20:03:16 on dist-test-slave-p3t5"
I20260812 20:03:16.274200 15159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "2f24c96fdace4b1b88eebe2c7a1c2ba4"
format_stamp: "Formatted at 2026-08-12 20:03:16 on dist-test-slave-p3t5"
I20260812 20:03:16.280470 15159 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 20:03:16.284734 15179 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:16.286526 15159 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:16.286778 15159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "2f24c96fdace4b1b88eebe2c7a1c2ba4"
format_stamp: "Formatted at 2026-08-12 20:03:16 on dist-test-slave-p3t5"
I20260812 20:03:16.287034 15159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:16.304718 15159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:16.309580 15159 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:16.309999 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:16.311208 15159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:16.313028 15159 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:16.313117 15159 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:16.313275 15159 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:16.313372 15159 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:16.350425 15159 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:38045
I20260812 20:03:16.350529 15307 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:38045 every 8 connection(s)
I20260812 20:03:16.352044 15159 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:16.353226 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15159
I20260812 20:03:16.353461 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:16.375285 15308 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40861
I20260812 20:03:16.375694 15308 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:16.376608 15308 heartbeater.cc:507] Master 127.25.14.62:40861 requested a full tablet report, sending...
I20260812 20:03:16.379077 15088 ts_manager.cc:194] Registered new tserver with Master: 2f24c96fdace4b1b88eebe2c7a1c2ba4 (127.25.14.1:38045)
I20260812 20:03:16.380800 15088 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:53877
I20260812 20:03:16.383675 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:16.466230 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15159
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:16.484083 15303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:16.576149 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15159
I20260812 20:03:16.601804 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15051
I20260812 20:03:16.603093 15120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:16.716289 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15051
2026-08-12T20:03:16Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3 (4458 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15324](info): set up 2 sockets
Aug 12 20:03:16 dist-test-slave-p3t5 krb5kdc[15324](info): commencing operation
krb5kdc: starting...
W20260812 20:03:18.980225 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.212s user 0.004s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:19 dist-test-slave-p3t5 krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:19.486171 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36133
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36133
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:19.877832 15347 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:19.878201 15347 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:19.878301 15347 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:19.886986 15347 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:19.887104 15347 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:19.887171 15347 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:19.887223 15347 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:19.898515 15347 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36133
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:19.900331 15347 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:19.901813 15347 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:19.910995 15352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:19.911041 15353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:19.911866 15357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:19.912593 15347 server_base.cc:1061] running on GCE node
I20260812 20:03:19.913645 15347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:19.915510 15347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:19.916739 15347 hybrid_clock.cc:648] HybridClock initialized: now 1786564999916699 us; error 37 us; skew 500 ppm
I20260812 20:03:19.917116 15347 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:19 dist-test-slave-p3t5 krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564999, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:19.920650 15347 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:19.922837 15347 webserver.cc:533] Webserver started at http://127.25.14.62:46391/ using document root <none> and password file <none>
I20260812 20:03:19.923671 15347 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:19.923818 15347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:19.924221 15347 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:19.926926 15347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "66ac33f5497c40e7adc84aaecdddc06a"
format_stamp: "Formatted at 2026-08-12 20:03:19 on dist-test-slave-p3t5"
I20260812 20:03:19.927819 15347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "66ac33f5497c40e7adc84aaecdddc06a"
format_stamp: "Formatted at 2026-08-12 20:03:19 on dist-test-slave-p3t5"
I20260812 20:03:19.934012 15347 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:03:19.938465 15364 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:19.940248 15347 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:03:19.940488 15347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "66ac33f5497c40e7adc84aaecdddc06a"
format_stamp: "Formatted at 2026-08-12 20:03:19 on dist-test-slave-p3t5"
I20260812 20:03:19.940744 15347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:19.978024 15347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:19.982915 15347 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:19.983274 15347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:20.006129 15347 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:36133
I20260812 20:03:20.006163 15428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:36133 every 8 connection(s)
I20260812 20:03:20.007835 15347 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:20.010012 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15347
I20260812 20:03:20.010205 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:20.013904 15429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786564999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:20.024989 15429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a: Bootstrap starting.
I20260812 20:03:20.029110 15429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:20.030406 15429 log.cc:826] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:20.033701 15429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a: No bootstrap required, opened a new log
I20260812 20:03:20.039953 15429 raft_consensus.cc:359] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } }
I20260812 20:03:20.040382 15429 raft_consensus.cc:385] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:20.040530 15429 raft_consensus.cc:740] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ac33f5497c40e7adc84aaecdddc06a, State: Initialized, Role: FOLLOWER
I20260812 20:03:20.041199 15429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } }
I20260812 20:03:20.041450 15429 raft_consensus.cc:399] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:20.041647 15429 raft_consensus.cc:493] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:20.041860 15429 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:20.044174 15429 raft_consensus.cc:515] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } }
I20260812 20:03:20.044791 15429 leader_election.cc:304] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ac33f5497c40e7adc84aaecdddc06a; no voters:
I20260812 20:03:20.045542 15429 leader_election.cc:290] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:20.045718 15435 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:20.047219 15435 raft_consensus.cc:697] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [term 1 LEADER]: Becoming Leader. State: Replica: 66ac33f5497c40e7adc84aaecdddc06a, State: Running, Role: LEADER
I20260812 20:03:20.047820 15435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } }
I20260812 20:03:20.048246 15429 sys_catalog.cc:565] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:20.050977 15437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ac33f5497c40e7adc84aaecdddc06a. Latest consensus state: current_term: 1 leader_uuid: "66ac33f5497c40e7adc84aaecdddc06a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } } }
I20260812 20:03:20.051304 15437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:20.051865 15436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ac33f5497c40e7adc84aaecdddc06a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ac33f5497c40e7adc84aaecdddc06a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36133 } } }
I20260812 20:03:20.052260 15436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:20.054867 15443 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:20.061035 15443 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:20.071359 15443 catalog_manager.cc:1383] Generated new cluster ID: 2dd93746298847528344d4632d1bc3ff
I20260812 20:03:20.071627 15443 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:20.089208 15443 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:20.090736 15443 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:20.107146 15443 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 66ac33f5497c40e7adc84aaecdddc06a: Generated new TSK 0
I20260812 20:03:20.108548 15443 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:20.295711 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:36133
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:20.698298 15461 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:20.698762 15461 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:20.698932 15461 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:20.707880 15461 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:20.708161 15461 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:20.719864 15461 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:20.721832 15461 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:20.723399 15461 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:20.734609 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:20.735028 15468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:20.735145 15470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:20.735909 15461 server_base.cc:1061] running on GCE node
I20260812 20:03:20.736573 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:20.737705 15461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:20.739045 15461 hybrid_clock.cc:648] HybridClock initialized: now 1786565000738928 us; error 108 us; skew 500 ppm
I20260812 20:03:20.739384 15461 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565000, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:20.742941 15461 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:20.744720 15461 webserver.cc:533] Webserver started at http://127.25.14.1:36849/ using document root <none> and password file <none>
I20260812 20:03:20.745497 15461 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:20.745602 15461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:20.746049 15461 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:20.748873 15461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "42562f07edc3445bb941a7ec333d6ed6"
format_stamp: "Formatted at 2026-08-12 20:03:20 on dist-test-slave-p3t5"
I20260812 20:03:20.749904 15461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "42562f07edc3445bb941a7ec333d6ed6"
format_stamp: "Formatted at 2026-08-12 20:03:20 on dist-test-slave-p3t5"
I20260812 20:03:20.756227 15461 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:03:20.760372 15478 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:20.762110 15461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:20.762341 15461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "42562f07edc3445bb941a7ec333d6ed6"
format_stamp: "Formatted at 2026-08-12 20:03:20 on dist-test-slave-p3t5"
I20260812 20:03:20.762573 15461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:20.780341 15461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:20.788499 15461 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:20.788967 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:20.790252 15461 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:20.792027 15461 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:20.792147 15461 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:20.792342 15461 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:20.792425 15461 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:20.833166 15461 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35853
I20260812 20:03:20.833216 15608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35853 every 8 connection(s)
I20260812 20:03:20.834798 15461 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:20.840668 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15461
I20260812 20:03:20.840864 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565000, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:20.857741 15609 heartbeater.cc:344] Connected to a master server at 127.25.14.62:36133
I20260812 20:03:20.858196 15609 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:20.859156 15609 heartbeater.cc:507] Master 127.25.14.62:36133 requested a full tablet report, sending...
I20260812 20:03:20.861847 15383 ts_manager.cc:194] Registered new tserver with Master: 42562f07edc3445bb941a7ec333d6ed6 (127.25.14.1:35853)
I20260812 20:03:20.863528 15383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:57031
I20260812 20:03:20.870464 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:20.953940 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15461
Aug 12 20:03:20 dist-test-slave-p3t5 krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:20.971266 15604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:21.061602 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15461
I20260812 20:03:21.087154 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15347
I20260812 20:03:21.088574 15422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:21.198671 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15347
2026-08-12T20:03:21Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4 (4486 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:21 dist-test-slave-p3t5 krb5kdc[15627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:21 dist-test-slave-p3t5 krb5kdc[15627](info): set up 2 sockets
Aug 12 20:03:21 dist-test-slave-p3t5 krb5kdc[15627](info): commencing operation
krb5kdc: starting...
W20260812 20:03:23.492588 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.238s user 0.001s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:23 dist-test-slave-p3t5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:23.993356 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40719
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40719
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:24.389981 15649 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:24.390369 15649 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:24.390460 15649 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:24.399399 15649 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:24.399528 15649 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:24.399600 15649 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:24.399643 15649 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:24.411473 15649 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40719
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:24.413264 15649 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:24.414706 15649 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:24.424295 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:24.424275 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:24.425304 15657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:24.425666 15649 server_base.cc:1061] running on GCE node
I20260812 20:03:24.426522 15649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:24.428058 15649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:24.429288 15649 hybrid_clock.cc:648] HybridClock initialized: now 1786565004429232 us; error 49 us; skew 500 ppm
I20260812 20:03:24.429776 15649 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:24 dist-test-slave-p3t5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565004, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:24.433209 15649 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:24.435144 15649 webserver.cc:533] Webserver started at http://127.25.14.62:33603/ using document root <none> and password file <none>
I20260812 20:03:24.435907 15649 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:24.436022 15649 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:24.436363 15649 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:24.439005 15649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "5934554a610c40348d28e7d683259c42"
format_stamp: "Formatted at 2026-08-12 20:03:24 on dist-test-slave-p3t5"
I20260812 20:03:24.439785 15649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "5934554a610c40348d28e7d683259c42"
format_stamp: "Formatted at 2026-08-12 20:03:24 on dist-test-slave-p3t5"
I20260812 20:03:24.445845 15649 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:03:24.449899 15664 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:24.451449 15649 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:03:24.451683 15649 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "5934554a610c40348d28e7d683259c42"
format_stamp: "Formatted at 2026-08-12 20:03:24 on dist-test-slave-p3t5"
I20260812 20:03:24.451907 15649 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:24.468396 15649 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:24.472656 15649 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:24.473030 15649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:24.495371 15649 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40719
I20260812 20:03:24.495434 15722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40719 every 8 connection(s)
I20260812 20:03:24.497303 15649 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:24.501962 15723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:24.506304 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15649
I20260812 20:03:24.506524 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:24.513713 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42: Bootstrap starting.
I20260812 20:03:24.517431 15723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:24.518930 15723 log.cc:826] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42: Log is configured to *not* fsync() on all Append() calls
Aug 12 20:03:24 dist-test-slave-p3t5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:24.522205 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42: No bootstrap required, opened a new log
I20260812 20:03:24.528573 15723 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } }
I20260812 20:03:24.529037 15723 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:24.529193 15723 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5934554a610c40348d28e7d683259c42, State: Initialized, Role: FOLLOWER
I20260812 20:03:24.530097 15723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } }
I20260812 20:03:24.530388 15723 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:24.530543 15723 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:24.530718 15723 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:24.533376 15723 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } }
I20260812 20:03:24.533946 15723 leader_election.cc:304] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5934554a610c40348d28e7d683259c42; no voters:
I20260812 20:03:24.534602 15723 leader_election.cc:290] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:24.534798 15729 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:24.536727 15729 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [term 1 LEADER]: Becoming Leader. State: Replica: 5934554a610c40348d28e7d683259c42, State: Running, Role: LEADER
I20260812 20:03:24.537222 15723 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:24.537352 15729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } }
I20260812 20:03:24.541091 15730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5934554a610c40348d28e7d683259c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } } }
I20260812 20:03:24.541235 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5934554a610c40348d28e7d683259c42. Latest consensus state: current_term: 1 leader_uuid: "5934554a610c40348d28e7d683259c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5934554a610c40348d28e7d683259c42" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40719 } } }
I20260812 20:03:24.541486 15730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:24.541582 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:24.542930 15739 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:24.554764 15739 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:24.569423 15739 catalog_manager.cc:1383] Generated new cluster ID: a4fddf7776a14a2daabbf46832154920
I20260812 20:03:24.569571 15739 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:24.581135 15739 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:24.582841 15739 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:24.599834 15739 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 5934554a610c40348d28e7d683259c42: Generated new TSK 0
I20260812 20:03:24.600827 15739 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:24.777482 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40719
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:25.172501 15756 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:25.172859 15756 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:25.172963 15756 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:25.181778 15756 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:25.181989 15756 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:25.193279 15756 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:40719
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:25.195122 15756 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:25.196532 15756 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:25.207247 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:25.207177 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:25.207743 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:25.208796 15756 server_base.cc:1061] running on GCE node
I20260812 20:03:25.209654 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:25.210822 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:25.212054 15756 hybrid_clock.cc:648] HybridClock initialized: now 1786565005211971 us; error 72 us; skew 500 ppm
I20260812 20:03:25.212445 15756 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565005, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:25.216082 15756 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:25.218156 15756 webserver.cc:533] Webserver started at http://127.25.14.1:37935/ using document root <none> and password file <none>
I20260812 20:03:25.219017 15756 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:25.219120 15756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:25.219569 15756 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:25.222357 15756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "fd5b633cdc3c4e1391b8ff80d1fe79fd"
format_stamp: "Formatted at 2026-08-12 20:03:25 on dist-test-slave-p3t5"
I20260812 20:03:25.223261 15756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "fd5b633cdc3c4e1391b8ff80d1fe79fd"
format_stamp: "Formatted at 2026-08-12 20:03:25 on dist-test-slave-p3t5"
I20260812 20:03:25.229789 15756 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 20:03:25.233785 15771 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:25.235391 15756 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 20:03:25.235620 15756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "fd5b633cdc3c4e1391b8ff80d1fe79fd"
format_stamp: "Formatted at 2026-08-12 20:03:25 on dist-test-slave-p3t5"
I20260812 20:03:25.235818 15756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:25.253863 15756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:25.259125 15756 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:25.259661 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:25.261004 15756 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:25.262980 15756 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:25.263079 15756 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:25.263221 15756 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:25.263311 15756 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:25.301282 15756 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:33657
I20260812 20:03:25.301385 15902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:33657 every 8 connection(s)
I20260812 20:03:25.302968 15756 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:25.311514 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15756
I20260812 20:03:25.311775 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565005, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:25.327481 15903 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40719
I20260812 20:03:25.327870 15903 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:25.328809 15903 heartbeater.cc:507] Master 127.25.14.62:40719 requested a full tablet report, sending...
I20260812 20:03:25.331473 15680 ts_manager.cc:194] Registered new tserver with Master: fd5b633cdc3c4e1391b8ff80d1fe79fd (127.25.14.1:33657)
I20260812 20:03:25.333348 15680 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:42375
I20260812 20:03:25.341460 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:25.427654 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15756
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:25.447798 15897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:25.535401 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15756
I20260812 20:03:25.560320 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15649
I20260812 20:03:25.561964 15718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:25.672300 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15649
2026-08-12T20:03:25Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5 (4469 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15917](info): set up 2 sockets
Aug 12 20:03:25 dist-test-slave-p3t5 krb5kdc[15917](info): commencing operation
krb5kdc: starting...
W20260812 20:03:27.986740 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.263s user 0.004s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:28 dist-test-slave-p3t5 krb5kdc[15917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:28.494228 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45575
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:45575
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:28.907447 15934 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:28.907902 15934 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:28.908082 15934 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:28.916882 15934 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:28.917069 15934 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:28.917214 15934 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:28.917300 15934 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:28.928654 15934 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:45575
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:28.930567 15934 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:28.932109 15934 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:28.940949 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:28.941056 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:28.942374 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:28.942864 15934 server_base.cc:1061] running on GCE node
I20260812 20:03:28.943786 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:28.945379 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:28.946627 15934 hybrid_clock.cc:648] HybridClock initialized: now 1786565008946571 us; error 49 us; skew 500 ppm
I20260812 20:03:28.947100 15934 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:28 dist-test-slave-p3t5 krb5kdc[15917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:28.950556 15934 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:28.952454 15934 webserver.cc:533] Webserver started at http://127.25.14.62:39951/ using document root <none> and password file <none>
I20260812 20:03:28.953289 15934 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:28.953457 15934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:28.953934 15934 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:28.956609 15934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "868cd4337cd84c44a3332dc828898553"
format_stamp: "Formatted at 2026-08-12 20:03:28 on dist-test-slave-p3t5"
I20260812 20:03:28.957538 15934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "868cd4337cd84c44a3332dc828898553"
format_stamp: "Formatted at 2026-08-12 20:03:28 on dist-test-slave-p3t5"
I20260812 20:03:28.963780 15934 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260812 20:03:28.968221 15949 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:28.969972 15934 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:28.970193 15934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "868cd4337cd84c44a3332dc828898553"
format_stamp: "Formatted at 2026-08-12 20:03:28 on dist-test-slave-p3t5"
I20260812 20:03:28.970453 15934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:28.997149 15934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:29.001541 15934 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:29.001899 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:29.025520 15934 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:45575
I20260812 20:03:29.025568 16007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:45575 every 8 connection(s)
I20260812 20:03:29.027344 15934 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:29.027673 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 15934
I20260812 20:03:29.027871 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:29.033376 16010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:29.044379 16010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553: Bootstrap starting.
I20260812 20:03:29.048434 16010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:29.049753 16010 log.cc:826] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:29.053118 16010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553: No bootstrap required, opened a new log
I20260812 20:03:29.059232 16010 raft_consensus.cc:359] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } }
I20260812 20:03:29.059628 16010 raft_consensus.cc:385] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:29.059723 16010 raft_consensus.cc:740] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868cd4337cd84c44a3332dc828898553, State: Initialized, Role: FOLLOWER
I20260812 20:03:29.060338 16010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } }
I20260812 20:03:29.060580 16010 raft_consensus.cc:399] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:29.060750 16010 raft_consensus.cc:493] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:29.060966 16010 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:29.063382 16010 raft_consensus.cc:515] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } }
I20260812 20:03:29.063953 16010 leader_election.cc:304] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 868cd4337cd84c44a3332dc828898553; no voters:
I20260812 20:03:29.064706 16010 leader_election.cc:290] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:29.064934 16015 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:29.065742 16015 raft_consensus.cc:697] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [term 1 LEADER]: Becoming Leader. State: Replica: 868cd4337cd84c44a3332dc828898553, State: Running, Role: LEADER
I20260812 20:03:29.066300 16015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } }
I20260812 20:03:29.067188 16010 sys_catalog.cc:565] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:29.069523 16016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "868cd4337cd84c44a3332dc828898553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } } }
I20260812 20:03:29.069902 16016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:29.069726 16017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 868cd4337cd84c44a3332dc828898553. Latest consensus state: current_term: 1 leader_uuid: "868cd4337cd84c44a3332dc828898553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868cd4337cd84c44a3332dc828898553" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 45575 } } }
I20260812 20:03:29.070147 16017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:29.072818 16021 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:29.078752 16021 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:29.089929 16021 catalog_manager.cc:1383] Generated new cluster ID: c5b76fd9a7774f27ad3b94f2c5116c88
I20260812 20:03:29.090096 16021 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:29.106482 16021 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:29.107802 16021 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:29.124670 16021 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 868cd4337cd84c44a3332dc828898553: Generated new TSK 0
I20260812 20:03:29.126032 16021 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:29.299506 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:45575
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:29.696025 16040 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:29.696416 16040 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:29.696619 16040 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:29.705606 16040 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:29.705917 16040 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:29.717310 16040 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:45575
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:29.719151 16040 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:29.720667 16040 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:29.731390 16050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:29.731448 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:29.732355 16054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:29.732722 16040 server_base.cc:1061] running on GCE node
I20260812 20:03:29.733415 16040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:29.734611 16040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:29.735916 16040 hybrid_clock.cc:648] HybridClock initialized: now 1786565009735842 us; error 60 us; skew 500 ppm
I20260812 20:03:29.736317 16040 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565009, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:29.739948 16040 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:29.742046 16040 webserver.cc:533] Webserver started at http://127.25.14.1:46183/ using document root <none> and password file <none>
I20260812 20:03:29.742890 16040 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:29.743038 16040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:29.743453 16040 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:29.746369 16040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "a232cb1d3ea841e0900612bde28c0a83"
format_stamp: "Formatted at 2026-08-12 20:03:29 on dist-test-slave-p3t5"
I20260812 20:03:29.747323 16040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "a232cb1d3ea841e0900612bde28c0a83"
format_stamp: "Formatted at 2026-08-12 20:03:29 on dist-test-slave-p3t5"
I20260812 20:03:29.753374 16040 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:29.757285 16061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:29.758805 16040 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:29.759037 16040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "a232cb1d3ea841e0900612bde28c0a83"
format_stamp: "Formatted at 2026-08-12 20:03:29 on dist-test-slave-p3t5"
I20260812 20:03:29.759248 16040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:29.776786 16040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:29.781097 16040 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:29.781436 16040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:29.782632 16040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:29.784497 16040 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:29.784585 16040 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:29.784718 16040 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:29.784806 16040 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:29.823264 16040 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:42131
I20260812 20:03:29.823375 16186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:42131 every 8 connection(s)
I20260812 20:03:29.824856 16040 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:29.834065 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16040
I20260812 20:03:29.834301 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565009, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:29.849565 16187 heartbeater.cc:344] Connected to a master server at 127.25.14.62:45575
I20260812 20:03:29.850203 16187 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:29.851475 16187 heartbeater.cc:507] Master 127.25.14.62:45575 requested a full tablet report, sending...
I20260812 20:03:29.854168 15969 ts_manager.cc:194] Registered new tserver with Master: a232cb1d3ea841e0900612bde28c0a83 (127.25.14.1:42131)
I20260812 20:03:29.855939 15969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:52203
I20260812 20:03:29.864598 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:29.949895 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16040
Aug 12 20:03:29 dist-test-slave-p3t5 krb5kdc[15917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:29.970307 16180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:30.060600 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16040
I20260812 20:03:30.086519 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 15934
I20260812 20:03:30.087853 16003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:30.196323 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 15934
2026-08-12T20:03:30Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6 (4523 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:30 dist-test-slave-p3t5 krb5kdc[16202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:30 dist-test-slave-p3t5 krb5kdc[16202](info): set up 2 sockets
Aug 12 20:03:30 dist-test-slave-p3t5 krb5kdc[16202](info): commencing operation
krb5kdc: starting...
W20260812 20:03:32.419229 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.172s user 0.004s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:32 dist-test-slave-p3t5 krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:32.937494 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43107
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:43107
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:33.330062 16220 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:33.330418 16220 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:33.330514 16220 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:33.339185 16220 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:33.339310 16220 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:33.339380 16220 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:33.339421 16220 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:33.350883 16220 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43107
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:33.352690 16220 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:33.354216 16220 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:33.363602 16229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:33.363610 16227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:33.364157 16231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:33.365501 16220 server_base.cc:1061] running on GCE node
I20260812 20:03:33.366480 16220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:33.368134 16220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:33.369401 16220 hybrid_clock.cc:648] HybridClock initialized: now 1786565013369341 us; error 77 us; skew 500 ppm
I20260812 20:03:33.369879 16220 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:33 dist-test-slave-p3t5 krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565013, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:33.373256 16220 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:33.375051 16220 webserver.cc:533] Webserver started at http://127.25.14.62:32957/ using document root <none> and password file <none>
I20260812 20:03:33.375836 16220 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:33.375943 16220 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:33.376279 16220 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:33.378940 16220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "37a24312127249f59432ac1a9c1d4d0e"
format_stamp: "Formatted at 2026-08-12 20:03:33 on dist-test-slave-p3t5"
I20260812 20:03:33.379825 16220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "37a24312127249f59432ac1a9c1d4d0e"
format_stamp: "Formatted at 2026-08-12 20:03:33 on dist-test-slave-p3t5"
I20260812 20:03:33.386215 16220 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:03:33.390247 16240 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:33.391858 16220 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 20:03:33.392086 16220 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "37a24312127249f59432ac1a9c1d4d0e"
format_stamp: "Formatted at 2026-08-12 20:03:33 on dist-test-slave-p3t5"
I20260812 20:03:33.392321 16220 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:33.428122 16220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:33.432595 16220 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:33.432951 16220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:33.456694 16220 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43107
I20260812 20:03:33.456729 16298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43107 every 8 connection(s)
I20260812 20:03:33.458525 16220 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:33.461206 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16220
I20260812 20:03:33.461454 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:33.463915 16299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:33.477913 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e: Bootstrap starting.
I20260812 20:03:33.481393 16299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e: Neither blocks nor log segments found. Creating new log.
Aug 12 20:03:33 dist-test-slave-p3t5 krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:33.482679 16299 log.cc:826] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:33.486832 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e: No bootstrap required, opened a new log
I20260812 20:03:33.494349 16299 raft_consensus.cc:359] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } }
I20260812 20:03:33.494774 16299 raft_consensus.cc:385] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:33.494933 16299 raft_consensus.cc:740] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a24312127249f59432ac1a9c1d4d0e, State: Initialized, Role: FOLLOWER
I20260812 20:03:33.495683 16299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } }
I20260812 20:03:33.495970 16299 raft_consensus.cc:399] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:33.496104 16299 raft_consensus.cc:493] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:33.496243 16299 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:33.498750 16299 raft_consensus.cc:515] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } }
I20260812 20:03:33.499284 16299 leader_election.cc:304] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a24312127249f59432ac1a9c1d4d0e; no voters:
I20260812 20:03:33.499954 16299 leader_election.cc:290] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:33.500171 16305 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:33.501370 16305 raft_consensus.cc:697] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [term 1 LEADER]: Becoming Leader. State: Replica: 37a24312127249f59432ac1a9c1d4d0e, State: Running, Role: LEADER
I20260812 20:03:33.502096 16305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } }
I20260812 20:03:33.502539 16299 sys_catalog.cc:565] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:33.504935 16308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a24312127249f59432ac1a9c1d4d0e. Latest consensus state: current_term: 1 leader_uuid: "37a24312127249f59432ac1a9c1d4d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } } }
I20260812 20:03:33.505245 16308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:33.505714 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a24312127249f59432ac1a9c1d4d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a24312127249f59432ac1a9c1d4d0e" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43107 } } }
I20260812 20:03:33.506078 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:33.510429 16313 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:33.516589 16313 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:33.526857 16313 catalog_manager.cc:1383] Generated new cluster ID: b19e5a83279d41ae9e8189b311f34b09
I20260812 20:03:33.527010 16313 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:33.541340 16313 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:33.542887 16313 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:33.555124 16313 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 37a24312127249f59432ac1a9c1d4d0e: Generated new TSK 0
I20260812 20:03:33.556273 16313 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:03:33.665302 25656 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.25.14.1: real 0.101s user 0.001s sys 0.035s
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:33.751878 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43107
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:34.145831 16334 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:34.146250 16334 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:34.146425 16334 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:34.155153 16334 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:34.155436 16334 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:34.166983 16334 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:34.168879 16334 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:34.170532 16334 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:34.181875 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:34.181813 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:34.181908 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:34.183351 16334 server_base.cc:1061] running on GCE node
I20260812 20:03:34.184043 16334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:34.185191 16334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:34.186460 16334 hybrid_clock.cc:648] HybridClock initialized: now 1786565014186405 us; error 48 us; skew 500 ppm
I20260812 20:03:34.186861 16334 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565014, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:34.190927 16334 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:34.192852 16334 webserver.cc:533] Webserver started at http://127.25.14.1:37099/ using document root <none> and password file <none>
I20260812 20:03:34.193784 16334 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:34.193940 16334 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:34.194352 16334 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:34.197065 16334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "babfe476afac4ac894fc1488c06bfbdc"
format_stamp: "Formatted at 2026-08-12 20:03:34 on dist-test-slave-p3t5"
I20260812 20:03:34.198136 16334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "babfe476afac4ac894fc1488c06bfbdc"
format_stamp: "Formatted at 2026-08-12 20:03:34 on dist-test-slave-p3t5"
I20260812 20:03:34.204607 16334 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 20:03:34.208920 16352 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:34.210724 16334 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:03:34.210997 16334 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "babfe476afac4ac894fc1488c06bfbdc"
format_stamp: "Formatted at 2026-08-12 20:03:34 on dist-test-slave-p3t5"
I20260812 20:03:34.211246 16334 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:34.239981 16334 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:34.244458 16334 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:34.244827 16334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:34.245983 16334 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:34.247797 16334 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:34.247910 16334 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:34.248070 16334 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:34.248142 16334 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:34.285041 16334 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39977
I20260812 20:03:34.285146 16484 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39977 every 8 connection(s)
I20260812 20:03:34.286799 16334 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:34.296309 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16334
I20260812 20:03:34.296530 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565014, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:34.310091 16485 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43107
I20260812 20:03:34.310470 16485 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:34.311662 16485 heartbeater.cc:507] Master 127.25.14.62:43107 requested a full tablet report, sending...
I20260812 20:03:34.314268 16261 ts_manager.cc:194] Registered new tserver with Master: babfe476afac4ac894fc1488c06bfbdc (127.25.14.1:39977)
I20260812 20:03:34.315977 16261 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:37625
I20260812 20:03:34.326223 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:34.408684 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16334
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:34.427753 16479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:34.527127 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16334
I20260812 20:03:34.553020 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16220
I20260812 20:03:34.554423 16294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:34.667363 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16220
2026-08-12T20:03:34Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7 (4474 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16501](info): set up 2 sockets
Aug 12 20:03:34 dist-test-slave-p3t5 krb5kdc[16501](info): commencing operation
krb5kdc: starting...
W20260812 20:03:36.947293 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.226s user 0.003s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:37 dist-test-slave-p3t5 krb5kdc[16501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:37.457695 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40667
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:40667
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:37.851518 16519 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:37.851943 16519 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:37.852121 16519 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:37.860760 16519 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:37.860891 16519 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:37.861003 16519 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:37.861096 16519 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:37.873086 16519 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:40667
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:37.875015 16519 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:37.876570 16519 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:37.886404 16524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:37.886633 16528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:37.886404 16526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:37.888109 16519 server_base.cc:1061] running on GCE node
I20260812 20:03:37.889014 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:37.890715 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:37.891959 16519 hybrid_clock.cc:648] HybridClock initialized: now 1786565017891895 us; error 51 us; skew 500 ppm
I20260812 20:03:37.892390 16519 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:37 dist-test-slave-p3t5 krb5kdc[16501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565017, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:37.896013 16519 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:37.897933 16519 webserver.cc:533] Webserver started at http://127.25.14.62:41355/ using document root <none> and password file <none>
I20260812 20:03:37.898849 16519 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:37.898964 16519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:37.899328 16519 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:37.902076 16519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "3e0beeb9444746aca687ca595836b54a"
format_stamp: "Formatted at 2026-08-12 20:03:37 on dist-test-slave-p3t5"
I20260812 20:03:37.902941 16519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "3e0beeb9444746aca687ca595836b54a"
format_stamp: "Formatted at 2026-08-12 20:03:37 on dist-test-slave-p3t5"
I20260812 20:03:37.909184 16519 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260812 20:03:37.913143 16537 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:37.914873 16519 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 20:03:37.915061 16519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "3e0beeb9444746aca687ca595836b54a"
format_stamp: "Formatted at 2026-08-12 20:03:37 on dist-test-slave-p3t5"
I20260812 20:03:37.915243 16519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:37.943333 16519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:37.947789 16519 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:37.948069 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:37.971206 16519 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:40667
I20260812 20:03:37.971328 16593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:40667 every 8 connection(s)
I20260812 20:03:37.972986 16519 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:37.978057 16594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:37.980726 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16519
I20260812 20:03:37.980943 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:37.990496 16594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a: Bootstrap starting.
I20260812 20:03:37.995362 16594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a: Neither blocks nor log segments found. Creating new log.
Aug 12 20:03:37 dist-test-slave-p3t5 krb5kdc[16501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:37.996709 16594 log.cc:826] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:38.000116 16594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a: No bootstrap required, opened a new log
I20260812 20:03:38.006388 16594 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } }
I20260812 20:03:38.006814 16594 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:38.006954 16594 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0beeb9444746aca687ca595836b54a, State: Initialized, Role: FOLLOWER
I20260812 20:03:38.007663 16594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } }
I20260812 20:03:38.007929 16594 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:38.008097 16594 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:38.008323 16594 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:38.010679 16594 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } }
I20260812 20:03:38.011168 16594 leader_election.cc:304] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e0beeb9444746aca687ca595836b54a; no voters:
I20260812 20:03:38.011902 16594 leader_election.cc:290] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:38.012102 16600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:38.013943 16600 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [term 1 LEADER]: Becoming Leader. State: Replica: 3e0beeb9444746aca687ca595836b54a, State: Running, Role: LEADER
I20260812 20:03:38.014631 16600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } }
I20260812 20:03:38.015357 16594 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:38.018193 16602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0beeb9444746aca687ca595836b54a. Latest consensus state: current_term: 1 leader_uuid: "3e0beeb9444746aca687ca595836b54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } } }
I20260812 20:03:38.018595 16602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:38.019730 16601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e0beeb9444746aca687ca595836b54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0beeb9444746aca687ca595836b54a" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 40667 } } }
I20260812 20:03:38.020048 16601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:38.035233 16609 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:38.040350 16609 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:38.050734 16609 catalog_manager.cc:1383] Generated new cluster ID: 1e3987171af24cc6becd4d4e5e25f1bc
I20260812 20:03:38.050901 16609 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:38.070327 16609 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:38.072252 16609 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:38.081888 16609 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 3e0beeb9444746aca687ca595836b54a: Generated new TSK 0
I20260812 20:03:38.083099 16609 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:38.263691 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:40667
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:38.698247 16625 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:38.698625 16625 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:38.698851 16625 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:38.708078 16625 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:38.708359 16625 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:38.720119 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:40667
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:38.722155 16625 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:38.723749 16625 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:38.735257 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:38.735213 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:38.735697 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:38.736228 16625 server_base.cc:1061] running on GCE node
I20260812 20:03:38.736950 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:38.738150 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:38.739388 16625 hybrid_clock.cc:648] HybridClock initialized: now 1786565018739330 us; error 43 us; skew 500 ppm
I20260812 20:03:38.739761 16625 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:38 dist-test-slave-p3t5 krb5kdc[16501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:38.743419 16625 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:38.745363 16625 webserver.cc:533] Webserver started at http://127.25.14.1:46819/ using document root <none> and password file <none>
I20260812 20:03:38.746285 16625 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:38.746448 16625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:38.746891 16625 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:38.749574 16625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "e0a28545accf48478015f55805ac71a7"
format_stamp: "Formatted at 2026-08-12 20:03:38 on dist-test-slave-p3t5"
I20260812 20:03:38.750571 16625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "e0a28545accf48478015f55805ac71a7"
format_stamp: "Formatted at 2026-08-12 20:03:38 on dist-test-slave-p3t5"
I20260812 20:03:38.756801 16625 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 20:03:38.761194 16643 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:38.763139 16625 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:38.763396 16625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "e0a28545accf48478015f55805ac71a7"
format_stamp: "Formatted at 2026-08-12 20:03:38 on dist-test-slave-p3t5"
I20260812 20:03:38.763676 16625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:38.786439 16625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:38.791527 16625 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:38.791953 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:38.793110 16625 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:38.794955 16625 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:38.795049 16625 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:38.795205 16625 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:38.795279 16625 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:38.832850 16625 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:36137
I20260812 20:03:38.832899 16772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:36137 every 8 connection(s)
I20260812 20:03:38.834519 16625 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:38.838980 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16625
I20260812 20:03:38.839182 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:38 dist-test-slave-p3t5 krb5kdc[16501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:38.856959 16773 heartbeater.cc:344] Connected to a master server at 127.25.14.62:40667
I20260812 20:03:38.857347 16773 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:38.858343 16773 heartbeater.cc:507] Master 127.25.14.62:40667 requested a full tablet report, sending...
I20260812 20:03:38.861127 16557 ts_manager.cc:194] Registered new tserver with Master: e0a28545accf48478015f55805ac71a7 (127.25.14.1:36137)
I20260812 20:03:38.863219 16557 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:32899
I20260812 20:03:38.868626 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:38 dist-test-slave-p3t5 krb5kdc[16501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:38 dist-test-slave-p3t5 krb5kdc[16501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:38.952778 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16625
Aug 12 20:03:38 dist-test-slave-p3t5 krb5kdc[16501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:38.972056 16768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:39.067570 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16625
I20260812 20:03:39.093456 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16519
I20260812 20:03:39.094880 16589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:39.206112 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16519
2026-08-12T20:03:39Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8 (4536 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:39 dist-test-slave-p3t5 krb5kdc[16788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:39 dist-test-slave-p3t5 krb5kdc[16788](info): set up 2 sockets
Aug 12 20:03:39 dist-test-slave-p3t5 krb5kdc[16788](info): commencing operation
krb5kdc: starting...
W20260812 20:03:41.468128 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.210s user 0.004s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:41 dist-test-slave-p3t5 krb5kdc[16788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:03:41.996912 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.62: real 0.100s user 0.018s sys 0.039s
I20260812 20:03:41.998011 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:44927
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:44927
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:42.385488 16805 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:42.385955 16805 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:42.386111 16805 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:42.395128 16805 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:42.395254 16805 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:42.395351 16805 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:42.395430 16805 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:42.406975 16805 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:44927
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:42.408900 16805 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:42.410624 16805 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:42.421968 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:42.422017 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:42.421911 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:42.427217 16805 server_base.cc:1061] running on GCE node
I20260812 20:03:42.428154 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:42.429854 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:42.431115 16805 hybrid_clock.cc:648] HybridClock initialized: now 1786565022431067 us; error 42 us; skew 500 ppm
I20260812 20:03:42.431538 16805 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:42 dist-test-slave-p3t5 krb5kdc[16788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565022, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:42.435060 16805 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:42.436996 16805 webserver.cc:533] Webserver started at http://127.25.14.62:33295/ using document root <none> and password file <none>
I20260812 20:03:42.437911 16805 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:42.438066 16805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:42.438464 16805 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:42.441112 16805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "dbe1b89fee36463dbd6bde8339bfd82b"
format_stamp: "Formatted at 2026-08-12 20:03:42 on dist-test-slave-p3t5"
I20260812 20:03:42.442106 16805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "dbe1b89fee36463dbd6bde8339bfd82b"
format_stamp: "Formatted at 2026-08-12 20:03:42 on dist-test-slave-p3t5"
I20260812 20:03:42.448442 16805 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:03:42.452739 16823 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:42.454562 16805 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:42.454808 16805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "dbe1b89fee36463dbd6bde8339bfd82b"
format_stamp: "Formatted at 2026-08-12 20:03:42 on dist-test-slave-p3t5"
I20260812 20:03:42.455050 16805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:42.475476 16805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:42.480224 16805 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:42.480576 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:42.503605 16805 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:44927
I20260812 20:03:42.503651 16883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:44927 every 8 connection(s)
I20260812 20:03:42.505313 16805 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:42.510439 16884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:42.511926 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16805
I20260812 20:03:42.512161 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:42.524163 16884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b: Bootstrap starting.
Aug 12 20:03:42 dist-test-slave-p3t5 krb5kdc[16788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:42.528323 16884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:42.529767 16884 log.cc:826] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:42.533420 16884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b: No bootstrap required, opened a new log
I20260812 20:03:42.540223 16884 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } }
I20260812 20:03:42.540653 16884 raft_consensus.cc:385] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:42.540772 16884 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe1b89fee36463dbd6bde8339bfd82b, State: Initialized, Role: FOLLOWER
I20260812 20:03:42.541517 16884 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } }
I20260812 20:03:42.541832 16884 raft_consensus.cc:399] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:42.541999 16884 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:42.542202 16884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:42.544572 16884 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } }
I20260812 20:03:42.545168 16884 leader_election.cc:304] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe1b89fee36463dbd6bde8339bfd82b; no voters:
I20260812 20:03:42.545924 16884 leader_election.cc:290] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:42.546079 16892 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:42.546934 16892 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [term 1 LEADER]: Becoming Leader. State: Replica: dbe1b89fee36463dbd6bde8339bfd82b, State: Running, Role: LEADER
I20260812 20:03:42.547503 16892 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } }
I20260812 20:03:42.548511 16884 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:42.551184 16894 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe1b89fee36463dbd6bde8339bfd82b. Latest consensus state: current_term: 1 leader_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } } }
I20260812 20:03:42.551630 16894 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:42.551661 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe1b89fee36463dbd6bde8339bfd82b" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 44927 } } }
I20260812 20:03:42.552043 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:42.555217 16899 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:42.562064 16899 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:42.575616 16899 catalog_manager.cc:1383] Generated new cluster ID: faad83165cf446bfa3d2488064fa2d0d
I20260812 20:03:42.575760 16899 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:42.582261 16911 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 20:03:42.586917 16899 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:42.588794 16899 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:42.601815 16899 catalog_manager.cc:6092] T 00000000000000000000000000000000 P dbe1b89fee36463dbd6bde8339bfd82b: Generated new TSK 0
I20260812 20:03:42.603145 16899 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:42.777285 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:44927
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:43.176352 16916 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:43.176699 16916 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:43.176795 16916 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:43.185765 16916 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:43.185990 16916 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:43.197367 16916 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:44927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:43.199208 16916 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:43.200698 16916 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:43.211257 16922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:43.211237 16921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:43.212280 16925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:43.213171 16916 server_base.cc:1061] running on GCE node
I20260812 20:03:43.213936 16916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:43.215085 16916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:43.216346 16916 hybrid_clock.cc:648] HybridClock initialized: now 1786565023216291 us; error 47 us; skew 500 ppm
I20260812 20:03:43.216715 16916 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[16788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:43.220494 16916 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:43.222496 16916 webserver.cc:533] Webserver started at http://127.25.14.1:44597/ using document root <none> and password file <none>
I20260812 20:03:43.223341 16916 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:43.223500 16916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:43.223918 16916 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:43.226748 16916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "4648c44a6ac545b39d658d6c893a00d3"
format_stamp: "Formatted at 2026-08-12 20:03:43 on dist-test-slave-p3t5"
I20260812 20:03:43.227689 16916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "4648c44a6ac545b39d658d6c893a00d3"
format_stamp: "Formatted at 2026-08-12 20:03:43 on dist-test-slave-p3t5"
I20260812 20:03:43.234375 16916 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 20:03:43.238816 16933 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:43.240571 16916 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 20:03:43.240834 16916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "4648c44a6ac545b39d658d6c893a00d3"
format_stamp: "Formatted at 2026-08-12 20:03:43 on dist-test-slave-p3t5"
I20260812 20:03:43.241108 16916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:43.268023 16916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:43.272622 16916 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:43.272974 16916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:43.274318 16916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:43.283195 16916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:43.283334 16916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:43.283499 16916 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:43.283579 16916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:43.323972 16916 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:37517
I20260812 20:03:43.324009 17065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:37517 every 8 connection(s)
I20260812 20:03:43.325528 16916 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:43.334079 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 16916
I20260812 20:03:43.334287 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[16788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:43.348394 17066 heartbeater.cc:344] Connected to a master server at 127.25.14.62:44927
I20260812 20:03:43.348963 17066 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:43.350345 17066 heartbeater.cc:507] Master 127.25.14.62:44927 requested a full tablet report, sending...
I20260812 20:03:43.352948 16843 ts_manager.cc:194] Registered new tserver with Master: 4648c44a6ac545b39d658d6c893a00d3 (127.25.14.1:37517)
I20260812 20:03:43.354686 16843 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:34893
I20260812 20:03:43.364609 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[16788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[16788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:43.450616 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16916
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[16788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:43.468214 17060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:43.556959 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16916
I20260812 20:03:43.582854 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 16805
I20260812 20:03:43.584214 16878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:43.695676 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 16805
2026-08-12T20:03:43Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9 (4489 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[17081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[17081](info): set up 2 sockets
Aug 12 20:03:43 dist-test-slave-p3t5 krb5kdc[17081](info): commencing operation
krb5kdc: starting...
W20260812 20:03:45.945152 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.197s user 0.001s sys 0.069s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:46 dist-test-slave-p3t5 krb5kdc[17081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:46.454079 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36501
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:36501
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:46.854562 17097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:46.854995 17097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:46.855173 17097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:46.863969 17097 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:46.864094 17097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:46.864200 17097 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:46.864298 17097 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:46.875698 17097 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:36501
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:46.877600 17097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:46.879201 17097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:46.889104 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:46.888743 17105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:46.889577 17104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:46.890514 17097 server_base.cc:1061] running on GCE node
I20260812 20:03:46.891464 17097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:46.893124 17097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:46.894405 17097 hybrid_clock.cc:648] HybridClock initialized: now 1786565026894348 us; error 51 us; skew 500 ppm
I20260812 20:03:46.894850 17097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:46 dist-test-slave-p3t5 krb5kdc[17081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565026, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:46.898649 17097 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:46.900566 17097 webserver.cc:533] Webserver started at http://127.25.14.62:39653/ using document root <none> and password file <none>
I20260812 20:03:46.901432 17097 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:46.901588 17097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:46.902050 17097 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:46.904994 17097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "256cc1abf0054748882a42cbe3916ee4"
format_stamp: "Formatted at 2026-08-12 20:03:46 on dist-test-slave-p3t5"
I20260812 20:03:46.906291 17097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "256cc1abf0054748882a42cbe3916ee4"
format_stamp: "Formatted at 2026-08-12 20:03:46 on dist-test-slave-p3t5"
I20260812 20:03:46.912650 17097 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:03:46.916762 17115 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:46.918445 17097 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:46.918671 17097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "256cc1abf0054748882a42cbe3916ee4"
format_stamp: "Formatted at 2026-08-12 20:03:46 on dist-test-slave-p3t5"
I20260812 20:03:46.918898 17097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:46.944682 17097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:46.949376 17097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:46.949753 17097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:46.975289 17097 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:36501
I20260812 20:03:46.975556 17173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:36501 every 8 connection(s)
I20260812 20:03:46.977020 17097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:46.978725 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17097
I20260812 20:03:46.978933 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:46.982738 17174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 20:03:46 dist-test-slave-p3t5 krb5kdc[17081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:46.997251 17174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4: Bootstrap starting.
I20260812 20:03:47.001940 17174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:47.003425 17174 log.cc:826] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:47.007261 17174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4: No bootstrap required, opened a new log
I20260812 20:03:47.013439 17174 raft_consensus.cc:359] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } }
I20260812 20:03:47.014041 17174 raft_consensus.cc:385] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:47.014223 17174 raft_consensus.cc:740] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256cc1abf0054748882a42cbe3916ee4, State: Initialized, Role: FOLLOWER
I20260812 20:03:47.014937 17174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } }
I20260812 20:03:47.015175 17174 raft_consensus.cc:399] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:47.015302 17174 raft_consensus.cc:493] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:47.015445 17174 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:47.018088 17174 raft_consensus.cc:515] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } }
I20260812 20:03:47.018816 17174 leader_election.cc:304] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256cc1abf0054748882a42cbe3916ee4; no voters:
I20260812 20:03:47.020006 17174 leader_election.cc:290] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:47.020255 17180 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:47.021150 17180 raft_consensus.cc:697] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [term 1 LEADER]: Becoming Leader. State: Replica: 256cc1abf0054748882a42cbe3916ee4, State: Running, Role: LEADER
I20260812 20:03:47.021867 17180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } }
I20260812 20:03:47.022913 17174 sys_catalog.cc:565] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:47.025261 17182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256cc1abf0054748882a42cbe3916ee4. Latest consensus state: current_term: 1 leader_uuid: "256cc1abf0054748882a42cbe3916ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } } }
I20260812 20:03:47.025736 17182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:47.025332 17181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256cc1abf0054748882a42cbe3916ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256cc1abf0054748882a42cbe3916ee4" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 36501 } } }
I20260812 20:03:47.029866 17181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:47.030566 17186 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:47.036813 17186 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:47.047613 17186 catalog_manager.cc:1383] Generated new cluster ID: 3709ff2194314117ae5779aacb04b24d
I20260812 20:03:47.047781 17186 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:47.077570 17186 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:47.079515 17186 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:47.092509 17186 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 256cc1abf0054748882a42cbe3916ee4: Generated new TSK 0
I20260812 20:03:47.093786 17186 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:47.277266 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:36501
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:47.687697 17206 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:47.688171 17206 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:47.688287 17206 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:47.697395 17206 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:47.697671 17206 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:47.709209 17206 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:36501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:47.711169 17206 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:47.712644 17206 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:47.724200 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:47.724757 17211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:47.725057 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:47.726260 17206 server_base.cc:1061] running on GCE node
I20260812 20:03:47.727051 17206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:47.728202 17206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:47.729501 17206 hybrid_clock.cc:648] HybridClock initialized: now 1786565027729401 us; error 79 us; skew 500 ppm
I20260812 20:03:47.729933 17206 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:47 dist-test-slave-p3t5 krb5kdc[17081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:47.733711 17206 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:47.735847 17206 webserver.cc:533] Webserver started at http://127.25.14.1:32805/ using document root <none> and password file <none>
I20260812 20:03:47.736739 17206 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:47.736886 17206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:47.737288 17206 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:47.740069 17206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "9c1910d7510b49ed895d55f0c2127ed0"
format_stamp: "Formatted at 2026-08-12 20:03:47 on dist-test-slave-p3t5"
I20260812 20:03:47.741024 17206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "9c1910d7510b49ed895d55f0c2127ed0"
format_stamp: "Formatted at 2026-08-12 20:03:47 on dist-test-slave-p3t5"
I20260812 20:03:47.747648 17206 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 20:03:47.751766 17223 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:47.753373 17206 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 20:03:47.753607 17206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "9c1910d7510b49ed895d55f0c2127ed0"
format_stamp: "Formatted at 2026-08-12 20:03:47 on dist-test-slave-p3t5"
I20260812 20:03:47.753923 17206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:47.779209 17206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:47.783735 17206 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:47.784101 17206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:47.785230 17206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:47.787122 17206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:47.787247 17206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:47.787420 17206 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:47.787499 17206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:47.826912 17206 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:39285
I20260812 20:03:47.827018 17343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:39285 every 8 connection(s)
I20260812 20:03:47.828552 17206 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:47.834148 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17206
I20260812 20:03:47.834388 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:47 dist-test-slave-p3t5 krb5kdc[17081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:47.852880 17344 heartbeater.cc:344] Connected to a master server at 127.25.14.62:36501
I20260812 20:03:47.853366 17344 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:47.854482 17344 heartbeater.cc:507] Master 127.25.14.62:36501 requested a full tablet report, sending...
I20260812 20:03:47.857394 17135 ts_manager.cc:194] Registered new tserver with Master: 9c1910d7510b49ed895d55f0c2127ed0 (127.25.14.1:39285)
I20260812 20:03:47.859400 17135 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:44719
I20260812 20:03:47.864235 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:47 dist-test-slave-p3t5 krb5kdc[17081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:47 dist-test-slave-p3t5 krb5kdc[17081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:47.948961 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17206
Aug 12 20:03:47 dist-test-slave-p3t5 krb5kdc[17081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:47.970552 17339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:48.061486 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17206
I20260812 20:03:48.086189 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17097
I20260812 20:03:48.087653 17169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:48.198086 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17097
2026-08-12T20:03:48Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10 (4503 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11
Loading random data
Initializing database '/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 20:03:48 dist-test-slave-p3t5 krb5kdc[17362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 20:03:48 dist-test-slave-p3t5 krb5kdc[17362](info): set up 2 sockets
Aug 12 20:03:48 dist-test-slave-p3t5 krb5kdc[17362](info): commencing operation
krb5kdc: starting...
W20260812 20:03:50.465893 25656 mini_kdc.cc:121] Time spent starting KDC: real 2.214s user 0.004s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/test-user.keytab.
Aug 12 20:03:50 dist-test-slave-p3t5 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T20:03:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
I20260812 20:03:50.983866 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43319
--webserver_interface=127.25.14.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:43319
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:51.380007 17381 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:51.380430 17381 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:51.380609 17381 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:51.389575 17381 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:51.389724 17381 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:51.389840 17381 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:51.389943 17381 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 20:03:51.401938 17381 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:43319
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:43319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:51.403825 17381 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:51.405368 17381 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:51.415071 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:51.415041 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:51.416149 17391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:51.416678 17381 server_base.cc:1061] running on GCE node
I20260812 20:03:51.417606 17381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:51.419395 17381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:51.420632 17381 hybrid_clock.cc:648] HybridClock initialized: now 1786565031420575 us; error 47 us; skew 500 ppm
I20260812 20:03:51.421059 17381 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:51 dist-test-slave-p3t5 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:51.424638 17381 init.cc:377] Logged in from keytab as kudu/127.25.14.62@KRBTEST.COM (short username kudu)
I20260812 20:03:51.426602 17381 webserver.cc:533] Webserver started at http://127.25.14.62:34767/ using document root <none> and password file <none>
I20260812 20:03:51.427512 17381 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:51.427668 17381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:51.428066 17381 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:51.430856 17381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "e1d305020e1d48eabd261c3aaa09870c"
format_stamp: "Formatted at 2026-08-12 20:03:51 on dist-test-slave-p3t5"
I20260812 20:03:51.431803 17381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "e1d305020e1d48eabd261c3aaa09870c"
format_stamp: "Formatted at 2026-08-12 20:03:51 on dist-test-slave-p3t5"
I20260812 20:03:51.438141 17381 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 20:03:51.442435 17400 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:51.444356 17381 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 20:03:51.444629 17381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "e1d305020e1d48eabd261c3aaa09870c"
format_stamp: "Formatted at 2026-08-12 20:03:51 on dist-test-slave-p3t5"
I20260812 20:03:51.444907 17381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:51.462734 17381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:51.472749 17381 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:51.473124 17381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:51.496176 17381 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:43319
I20260812 20:03:51.496197 17454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:43319 every 8 connection(s)
I20260812 20:03:51.497956 17381 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:51.502978 17455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:51.507937 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17381
I20260812 20:03:51.508169 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:51.514608 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c: Bootstrap starting.
I20260812 20:03:51.518477 17455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:51.519831 17455 log.cc:826] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c: Log is configured to *not* fsync() on all Append() calls
Aug 12 20:03:51 dist-test-slave-p3t5 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:51.523262 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c: No bootstrap required, opened a new log
I20260812 20:03:51.529605 17455 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } }
I20260812 20:03:51.530140 17455 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:51.530310 17455 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d305020e1d48eabd261c3aaa09870c, State: Initialized, Role: FOLLOWER
I20260812 20:03:51.531206 17455 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } }
I20260812 20:03:51.531495 17455 raft_consensus.cc:399] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 20:03:51.531653 17455 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 20:03:51.531852 17455 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:51.534673 17455 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } }
I20260812 20:03:51.535297 17455 leader_election.cc:304] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d305020e1d48eabd261c3aaa09870c; no voters:
I20260812 20:03:51.536041 17455 leader_election.cc:290] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 20:03:51.536223 17460 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:51.537015 17460 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [term 1 LEADER]: Becoming Leader. State: Replica: e1d305020e1d48eabd261c3aaa09870c, State: Running, Role: LEADER
I20260812 20:03:51.537580 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } }
I20260812 20:03:51.538522 17455 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:51.541813 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d305020e1d48eabd261c3aaa09870c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } } }
I20260812 20:03:51.541882 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d305020e1d48eabd261c3aaa09870c. Latest consensus state: current_term: 1 leader_uuid: "e1d305020e1d48eabd261c3aaa09870c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d305020e1d48eabd261c3aaa09870c" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 43319 } } }
I20260812 20:03:51.542339 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:51.542196 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:51.544629 17471 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:51.551159 17471 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:51.561728 17471 catalog_manager.cc:1383] Generated new cluster ID: 833cd0a09fa9463eb741ca61a1cf845b
I20260812 20:03:51.561975 17471 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:51.602142 17471 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:51.603588 17471 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:51.615015 17471 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e1d305020e1d48eabd261c3aaa09870c: Generated new TSK 0
I20260812 20:03:51.616047 17471 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.25.14.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.14.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.25.14.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab.
W20260812 20:03:51.817317 25656 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.25.14.1: real 0.105s user 0.003s sys 0.032s
I20260812 20:03:51.818267 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:43319
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.25.14.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 20:03:52.240226 17484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:52.240604 17484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:52.240700 17484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:52.249419 17484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:52.249691 17484 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:52.261060 17484 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/krb5kdc/kudu.keytab
--principal=kudu/127.25.14.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.25.14.62:43319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:52.262856 17484 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:52.264387 17484 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:52.275605 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:52.275628 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:52.276242 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:52.276647 17484 server_base.cc:1061] running on GCE node
I20260812 20:03:52.277345 17484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:52.278529 17484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:52.279799 17484 hybrid_clock.cc:648] HybridClock initialized: now 1786565032279740 us; error 40 us; skew 500 ppm
I20260812 20:03:52.280217 17484 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 20:03:52 dist-test-slave-p3t5 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565032, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 20:03:52.284118 17484 init.cc:377] Logged in from keytab as kudu/127.25.14.1@KRBTEST.COM (short username kudu)
I20260812 20:03:52.286072 17484 webserver.cc:533] Webserver started at http://127.25.14.1:39435/ using document root <none> and password file <none>
I20260812 20:03:52.286921 17484 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:52.287032 17484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:52.287413 17484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:52.290092 17484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "2edbb2d0e3d046c4b85daefc5be28ccf"
format_stamp: "Formatted at 2026-08-12 20:03:52 on dist-test-slave-p3t5"
I20260812 20:03:52.290917 17484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "2edbb2d0e3d046c4b85daefc5be28ccf"
format_stamp: "Formatted at 2026-08-12 20:03:52 on dist-test-slave-p3t5"
I20260812 20:03:52.296844 17484 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:52.300861 17503 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:52.302672 17484 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:52.302917 17484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "2edbb2d0e3d046c4b85daefc5be28ccf"
format_stamp: "Formatted at 2026-08-12 20:03:52 on dist-test-slave-p3t5"
I20260812 20:03:52.303236 17484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:52.320541 17484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:52.325085 17484 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:52.325441 17484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:52.326623 17484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:52.328351 17484 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:52.328433 17484 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:52.328584 17484 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:52.328648 17484 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:52.365537 17484 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:35467
I20260812 20:03:52.365587 17627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:35467 every 8 connection(s)
I20260812 20:03:52.367471 17484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:52.370337 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17484
I20260812 20:03:52.370549 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
Aug 12 20:03:52 dist-test-slave-p3t5 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565032, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.14.1@KRBTEST.COM for kudu/127.25.14.62@KRBTEST.COM
I20260812 20:03:52.390579 17628 heartbeater.cc:344] Connected to a master server at 127.25.14.62:43319
I20260812 20:03:52.391010 17628 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:52.391980 17628 heartbeater.cc:507] Master 127.25.14.62:43319 requested a full tablet report, sending...
I20260812 20:03:52.394449 17418 ts_manager.cc:194] Registered new tserver with Master: 2edbb2d0e3d046c4b85daefc5be28ccf (127.25.14.1:35467)
I20260812 20:03:52.396145 17418 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.14.1@KRBTEST.COM'} at 127.25.14.1:46021
I20260812 20:03:52.400111 25656 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Aug 12 20:03:52 dist-test-slave-p3t5 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 20:03:52 dist-test-slave-p3t5 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.25.14.62@KRBTEST.COM
I20260812 20:03:52.483758 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17484
Aug 12 20:03:52 dist-test-slave-p3t5 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786565032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.14.1@KRBTEST.COM
I20260812 20:03:52.502283 17623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:52.593346 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17484
I20260812 20:03:52.619894 25656 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu pid 17381
I20260812 20:03:52.621292 17450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 20:03:52.730299 25656 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu with pid 17381
2026-08-12T20:03:52Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11 (4532 ms)
[----------] 37 tests from ExternalMiniClusterTest (670320 ms total)
[----------] 1 test from Kudu3762Test
[ RUN ] Kudu3762Test.AddMaster
2026-08-12T20:03:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T20:03:52Z Disabled control of system clock
I20260812 20:03:52.852864 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38865
--webserver_interface=127.25.14.62
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38865,127.25.14.61:37227
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
W20260812 20:03:53.261756 17645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:53.262225 17645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:53.262384 17645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:53.271358 17645 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 20:03:53.271477 17645 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260812 20:03:53.271615 17645 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:53.271705 17645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:53.271823 17645 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:53.271917 17645 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 20:03:53.272083 17645 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 20:03:53.272133 17645 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260812 20:03:53.283954 17645 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38865,127.25.14.61:37227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.62:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.14.62
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
W20260812 20:03:53.284439 17645 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.14.62:38865,127.25.14.61:37227'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 20:03:53.286095 17645 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:53.288751 17645 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:53.298759 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:53.298789 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:53.299892 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:53.301218 17645 server_base.cc:1061] running on GCE node
I20260812 20:03:53.302290 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:53.304116 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:53.305377 17645 hybrid_clock.cc:648] HybridClock initialized: now 1786565033305306 us; error 45 us; skew 500 ppm
I20260812 20:03:53.305941 17645 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:03:53.308565 17645 webserver.cc:533] Webserver started at http://127.25.14.62:38409/ using document root <none> and password file <none>
I20260812 20:03:53.309446 17645 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:53.309610 17645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:53.310084 17645 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:53.312916 17645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data/instance:
uuid: "62a275fb48f042dda1fbd49607d55fdb"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.313942 17645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal/instance:
uuid: "62a275fb48f042dda1fbd49607d55fdb"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.320782 17645 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 20:03:53.325019 17662 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:53.326692 17645 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:53.326974 17645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal
uuid: "62a275fb48f042dda1fbd49607d55fdb"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.327229 17645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:53.352421 17645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:53.353484 17645 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:53.353871 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:53.378638 17645 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.62:38865
I20260812 20:03:53.378677 17727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.62:38865 every 8 connection(s)
I20260812 20:03:53.380448 17645 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/data/info.pb
I20260812 20:03:53.385511 17728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:53.386497 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17645
I20260812 20:03:53.386778 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-0/wal/instance
I20260812 20:03:53.388283 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:37227
--webserver_interface=127.25.14.61
--webserver_port=0
--builtin_ntp_servers=127.25.14.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.14.62:38865,127.25.14.61:37227
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
I20260812 20:03:53.392933 17728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:03:53.412642 17728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } has no permanent_uuid. Determining permanent_uuid...
W20260812 20:03:53.414151 17663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
W20260812 20:03:53.416121 17728 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37227: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
I20260812 20:03:53.465513 17728 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } attempt: 1
W20260812 20:03:53.468142 17728 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37227: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
I20260812 20:03:53.536448 17728 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } attempt: 2
W20260812 20:03:53.538807 17728 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37227: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
I20260812 20:03:53.630131 17728 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } attempt: 3
W20260812 20:03:53.632651 17728 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37227: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
I20260812 20:03:53.775944 17728 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } attempt: 4
W20260812 20:03:53.778337 17728 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.14.61:37227: Network error: Client connection negotiation failed: client connection to 127.25.14.61:37227: connect: Connection refused (error 111)
W20260812 20:03:53.784504 17733 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:53.784953 17733 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:53.785101 17733 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:53.794137 17733 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 20:03:53.794283 17733 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260812 20:03:53.794435 17733 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 20:03:53.794523 17733 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:53.794637 17733 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 20:03:53.794728 17733 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 20:03:53.794876 17733 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 20:03:53.794950 17733 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260812 20:03:53.805958 17733 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.14.62:38865,127.25.14.61:37227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.14.61:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.14.61
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
W20260812 20:03:53.806437 17733 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.14.62:38865,127.25.14.61:37227'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 20:03:53.808182 17733 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:53.810446 17733 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:53.820192 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:53.820302 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:53.820192 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:53.821208 17733 server_base.cc:1061] running on GCE node
I20260812 20:03:53.822275 17733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:53.824075 17733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:53.825438 17733 hybrid_clock.cc:648] HybridClock initialized: now 1786565033825321 us; error 115 us; skew 500 ppm
I20260812 20:03:53.825928 17733 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:03:53.828516 17733 webserver.cc:533] Webserver started at http://127.25.14.61:39513/ using document root <none> and password file <none>
I20260812 20:03:53.829408 17733 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:53.829542 17733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:53.829957 17733 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:53.832718 17733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data/instance:
uuid: "a3cb6cd752da451f99894b4d53d113d1"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.833608 17733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal/instance:
uuid: "a3cb6cd752da451f99894b4d53d113d1"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.839653 17733 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 20:03:53.843854 17749 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:53.845422 17733 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 20:03:53.845718 17733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal
uuid: "a3cb6cd752da451f99894b4d53d113d1"
format_stamp: "Formatted at 2026-08-12 20:03:53 on dist-test-slave-p3t5"
I20260812 20:03:53.845963 17733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:53.863361 17733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:53.864399 17733 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:53.864749 17733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:53.888404 17733 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.61:37227
I20260812 20:03:53.888404 17813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.61:37227 every 8 connection(s)
I20260812 20:03:53.890272 17733 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/data/info.pb
I20260812 20:03:53.894073 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17733
I20260812 20:03:53.894306 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/master-1/wal/instance
I20260812 20:03:53.895218 17814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:53.903577 17814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:03:53.916323 17814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } has no permanent_uuid. Determining permanent_uuid...
I20260812 20:03:53.927415 17814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Bootstrap starting.
I20260812 20:03:53.931412 17814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:53.932754 17814 log.cc:826] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:53.936137 17814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: No bootstrap required, opened a new log
I20260812 20:03:53.941759 17814 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:53.942140 17814 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:53.942266 17814 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cb6cd752da451f99894b4d53d113d1, State: Initialized, Role: FOLLOWER
I20260812 20:03:53.943032 17814 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:53.943986 17820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:53.944291 17820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:03:53.945310 17814 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 20:03:53.949389 17759 tablet.cc:2406] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 20:03:53.955685 17833 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:03:53.955825 17833 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:03:53.952983 17733 master_runner.cc:187] Error getting master registration for 127.25.14.62:38865: OK, instance_id { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" instance_seqno: 1786565033372206 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 20:03:53.959262 17733 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 20:03:54.077684 17728 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } attempt: 5
I20260812 20:03:54.090740 17728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Bootstrap starting.
I20260812 20:03:54.095364 17728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:54.096879 17728 log.cc:826] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:54.100390 17728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: No bootstrap required, opened a new log
I20260812 20:03:54.108539 17728 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:54.108973 17728 raft_consensus.cc:385] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:54.109097 17728 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a275fb48f042dda1fbd49607d55fdb, State: Initialized, Role: FOLLOWER
I20260812 20:03:54.109923 17728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:54.110918 17836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:54.111210 17836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:03:54.111553 17820 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:54.111845 17820 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:54.112773 17728 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 20:03:54.113402 17820 leader_election.cc:290] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62a275fb48f042dda1fbd49607d55fdb (127.25.14.62:38865)
W20260812 20:03:54.113921 17671 tablet.cc:2406] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:03:54.114477 17698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cb6cd752da451f99894b4d53d113d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a275fb48f042dda1fbd49607d55fdb" is_pre_election: true
I20260812 20:03:54.115120 17698 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cb6cd752da451f99894b4d53d113d1 in term 0.
I20260812 20:03:54.115933 17750 leader_election.cc:304] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 62a275fb48f042dda1fbd49607d55fdb, a3cb6cd752da451f99894b4d53d113d1; no voters:
I20260812 20:03:54.116480 17820 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:54.116645 17820 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:54.116744 17820 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:54.119247 17820 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:54.120088 17820 leader_election.cc:290] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [CANDIDATE]: Term 1 election: Requested vote from peers 62a275fb48f042dda1fbd49607d55fdb (127.25.14.62:38865)
I20260812 20:03:54.120946 17698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cb6cd752da451f99894b4d53d113d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a275fb48f042dda1fbd49607d55fdb"
I20260812 20:03:54.121218 17698 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:54.124166 17698 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cb6cd752da451f99894b4d53d113d1 in term 1.
I20260812 20:03:54.124976 17750 leader_election.cc:304] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 62a275fb48f042dda1fbd49607d55fdb, a3cb6cd752da451f99894b4d53d113d1; no voters:
I20260812 20:03:54.125372 17820 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:54.128669 17820 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [term 1 LEADER]: Becoming Leader. State: Replica: a3cb6cd752da451f99894b4d53d113d1, State: Running, Role: LEADER
I20260812 20:03:54.129331 17820 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } }
I20260812 20:03:54.137487 17845 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cb6cd752da451f99894b4d53d113d1. Latest consensus state: current_term: 1 leader_uuid: "a3cb6cd752da451f99894b4d53d113d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:54.137872 17845 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: This master's current role is: LEADER
W20260812 20:03:54.138916 17848 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 20:03:54.139158 17848 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 20:03:54.146617 17851 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 20:03:54.149317 17851 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 20:03:54.159443 17698 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [term 1 FOLLOWER]: Refusing update from remote peer a3cb6cd752da451f99894b4d53d113d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 20:03:54.160633 17845 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:54.171509 17836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cb6cd752da451f99894b4d53d113d1. Latest consensus state: current_term: 1 leader_uuid: "a3cb6cd752da451f99894b4d53d113d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:54.171769 17836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:03:54.183853 17820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3cb6cd752da451f99894b4d53d113d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:54.184224 17820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1 [sys.catalog]: This master's current role is: LEADER
I20260812 20:03:54.184741 17851 catalog_manager.cc:1383] Generated new cluster ID: 44f41ae55cf04875a308d8744c3d3658
I20260812 20:03:54.184857 17851 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 20:03:54.188472 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:54.188642 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:54.200214 17836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3cb6cd752da451f99894b4d53d113d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62a275fb48f042dda1fbd49607d55fdb" member_type: VOTER last_known_addr { host: "127.25.14.62" port: 38865 } } peers { permanent_uuid: "a3cb6cd752da451f99894b4d53d113d1" member_type: VOTER last_known_addr { host: "127.25.14.61" port: 37227 } } }
I20260812 20:03:54.200475 17836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 20:03:54.250139 17851 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 20:03:54.252230 17851 catalog_manager.cc:1540] Loading token signing keys...
I20260812 20:03:54.274235 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260812 20:03:54.274608 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260812 20:03:54.274930 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260812 20:03:54.275825 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.277858 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:54.278172 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:54.280118 17851 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Generated new TSK 0
I20260812 20:03:54.281352 17851 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 20:03:54.281193 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.091s user 0.034s sys 0.011s Metrics: {"bytes_written":4776,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":826,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1315,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1}
I20260812 20:03:54.284276 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:54.294128 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:54.294492 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260812 20:03:54.294885 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260812 20:03:54.296554 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.297561 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:54.292582 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.103s user 0.028s sys 0.018s Metrics: {"bytes_written":4776,"cfile_init":1,"dirs.queue_time_us":8286,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1747,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1729,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1,"thread_start_us":9686,"threads_started":1}
I20260812 20:03:54.304837 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.011s sys 0.000s Metrics: {"bytes_written":5966,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1611,"drs_written":1,"lbm_read_time_us":195,"lbm_reads_lt_1ms":4,"lbm_write_time_us":980,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":2}
I20260812 20:03:54.306504 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.307168 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260812 20:03:54.307446 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.1:0
--local_ip_for_outbound_sockets=127.25.14.1
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--builtin_ntp_servers=127.25.14.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 20:03:54.307461 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260812 20:03:54.307724 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.310233 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.310773 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:54.311049 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":5966,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1832,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1021,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":2}
I20260812 20:03:54.314960 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260812 20:03:54.315376 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.315827 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.315986 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.316697 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.317327 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260812 20:03:54.364248 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 20:03:54.364625 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 20:03:54.365583 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.366070 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:54.368407 17756 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:54.377272 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.069s user 0.017s sys 0.023s Metrics: {"bytes_written":6026,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1552,"cfile_cache_miss":28,"cfile_cache_miss_bytes":2194,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1492,"dirs.run_cpu_time_us":684,"dirs.run_wall_time_us":8983,"drs_written":1,"lbm_read_time_us":1222,"lbm_reads_lt_1ms":56,"lbm_write_time_us":1119,"lbm_writes_lt_1ms":27,"mutex_wait_us":16,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":3,"thread_start_us":13216,"threads_started":1}
I20260812 20:03:54.392314 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.015s sys 0.024s Metrics: {"bytes_written":6026,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1552,"cfile_cache_miss":28,"cfile_cache_miss_bytes":2194,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":3186,"dirs.run_cpu_time_us":1031,"dirs.run_wall_time_us":4799,"drs_written":1,"lbm_read_time_us":1348,"lbm_reads_lt_1ms":56,"lbm_write_time_us":1233,"lbm_writes_lt_1ms":27,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":3,"spinlock_wait_cycles":319232,"thread_start_us":10606,"threads_started":2}
I20260812 20:03:54.393383 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 20:03:54.394209 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20260812 20:03:54.397797 17670 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:54.403488 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.077s user 0.011s sys 0.011s Metrics: {"dirs.queue_time_us":2343,"dirs.run_cpu_time_us":920,"dirs.run_wall_time_us":5723,"lbm_write_time_us":426,"lbm_writes_lt_1ms":7,"mutex_wait_us":130,"num_input_rowsets":0,"spinlock_wait_cycles":47921152,"thread_start_us":14842,"threads_started":1}
I20260812 20:03:54.404105 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.093s user 0.007s sys 0.005s Metrics: {"dirs.queue_time_us":5302,"dirs.run_cpu_time_us":1170,"dirs.run_wall_time_us":6887,"lbm_write_time_us":9706,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":6,"num_input_rowsets":0,"spinlock_wait_cycles":228608,"thread_start_us":38141,"threads_started":2}
I20260812 20:03:54.410903 17668 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:54.419782 17667 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:54.421389 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.097s user 0.009s sys 0.008s Metrics: {"dirs.queue_time_us":366,"dirs.run_cpu_time_us":749,"dirs.run_wall_time_us":3700,"lbm_write_time_us":381,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"spinlock_wait_cycles":40722304}
I20260812 20:03:54.428481 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.018s sys 0.004s Metrics: {"dirs.queue_time_us":3100,"dirs.run_cpu_time_us":934,"dirs.run_wall_time_us":4140,"lbm_write_time_us":372,"lbm_writes_lt_1ms":7,"mutex_wait_us":3845,"num_input_rowsets":0,"spinlock_wait_cycles":27257088}
W20260812 20:03:54.774014 17859 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:54.774389 17859 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:54.774531 17859 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:54.783808 17859 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:54.784065 17859 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.1
I20260812 20:03:54.795214 17859 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.14.1
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:54.797135 17859 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:54.799363 17859 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:54.811822 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:54.811913 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:54.812191 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:54.813459 17859 server_base.cc:1061] running on GCE node
I20260812 20:03:54.814280 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:54.815613 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:54.816851 17859 hybrid_clock.cc:648] HybridClock initialized: now 1786565034816804 us; error 41 us; skew 500 ppm
I20260812 20:03:54.817238 17859 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:03:54.819782 17859 webserver.cc:533] Webserver started at http://127.25.14.1:41545/ using document root <none> and password file <none>
I20260812 20:03:54.820705 17859 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:54.820811 17859 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:54.821164 17859 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:54.824110 17859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data/instance:
uuid: "f20226c1597d481a84f0c2fe53f0bb65"
format_stamp: "Formatted at 2026-08-12 20:03:54 on dist-test-slave-p3t5"
I20260812 20:03:54.825028 17859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance:
uuid: "f20226c1597d481a84f0c2fe53f0bb65"
format_stamp: "Formatted at 2026-08-12 20:03:54 on dist-test-slave-p3t5"
I20260812 20:03:54.831763 17859 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 20:03:54.836124 17882 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:54.837922 17859 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:54.838146 17859 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal
uuid: "f20226c1597d481a84f0c2fe53f0bb65"
format_stamp: "Formatted at 2026-08-12 20:03:54 on dist-test-slave-p3t5"
I20260812 20:03:54.838414 17859 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:54.869949 17859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:54.871053 17859 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:54.871402 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:54.872495 17859 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:54.874465 17859 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:54.874562 17859 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:54.874722 17859 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:54.874838 17859 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:54.917724 17859 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.1:46337
I20260812 20:03:54.917790 18015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.1:46337 every 8 connection(s)
I20260812 20:03:54.919430 17859 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/data/info.pb
I20260812 20:03:54.922572 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 17859
I20260812 20:03:54.922839 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/instance
I20260812 20:03:54.928112 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.2:0
--local_ip_for_outbound_sockets=127.25.14.2
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--builtin_ntp_servers=127.25.14.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 20:03:54.938217 18018 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38865
I20260812 20:03:54.938360 18016 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37227
I20260812 20:03:54.938715 18018 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:54.938717 18016 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:54.940163 18018 heartbeater.cc:507] Master 127.25.14.62:38865 requested a full tablet report, sending...
I20260812 20:03:54.940176 18016 heartbeater.cc:507] Master 127.25.14.61:37227 requested a full tablet report, sending...
I20260812 20:03:54.943084 17769 ts_manager.cc:194] Registered new tserver with Master: f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:03:54.943100 17683 ts_manager.cc:194] Registered new tserver with Master: f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:03:54.945086 17769 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.1:33347
I20260812 20:03:55.143123 17848 catalog_manager.cc:1295] Loaded cluster ID: 44f41ae55cf04875a308d8744c3d3658
I20260812 20:03:55.143270 17848 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: loading cluster ID for follower catalog manager: success
I20260812 20:03:55.147894 17848 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: acquiring CA information for follower catalog manager: success
I20260812 20:03:55.151068 17848 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 20:03:55.332374 18025 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:55.332743 18025 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:55.332871 18025 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:55.341800 18025 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:55.342061 18025 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.2
I20260812 20:03:55.353371 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.14.2
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:55.355190 18025 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:55.357188 18025 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:55.368322 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:55.368314 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:55.369061 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:55.369992 18025 server_base.cc:1061] running on GCE node
I20260812 20:03:55.370639 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:55.371671 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:55.372982 18025 hybrid_clock.cc:648] HybridClock initialized: now 1786565035372925 us; error 72 us; skew 500 ppm
I20260812 20:03:55.373381 18025 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:03:55.376034 18025 webserver.cc:533] Webserver started at http://127.25.14.2:42011/ using document root <none> and password file <none>
I20260812 20:03:55.376936 18025 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:55.377089 18025 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:55.377480 18025 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:55.381548 18025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data/instance:
uuid: "d222b63b24184ca6bc530a1547fe3297"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.382755 18025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance:
uuid: "d222b63b24184ca6bc530a1547fe3297"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.391256 18025 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260812 20:03:55.396389 18042 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:55.398190 18025 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 20:03:55.398379 18025 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal
uuid: "d222b63b24184ca6bc530a1547fe3297"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.398597 18025 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:55.443809 18025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:55.444938 18025 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:55.445240 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:55.446410 18025 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:55.448473 18025 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:55.448596 18025 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:55.448746 18025 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:55.448840 18025 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:55.487210 18025 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.2:34129
I20260812 20:03:55.487308 18177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.2:34129 every 8 connection(s)
I20260812 20:03:55.488965 18025 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/data/info.pb
I20260812 20:03:55.493636 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18025
I20260812 20:03:55.493860 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/instance
I20260812 20:03:55.503571 25656 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
/tmp/dist-test-taskQcgiFK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.14.3:0
--local_ip_for_outbound_sockets=127.25.14.3
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--builtin_ntp_servers=127.25.14.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260812 20:03:55.508169 18180 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38865
I20260812 20:03:55.508330 18178 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37227
I20260812 20:03:55.508659 18180 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:55.508667 18178 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:55.509804 18180 heartbeater.cc:507] Master 127.25.14.62:38865 requested a full tablet report, sending...
I20260812 20:03:55.509897 18178 heartbeater.cc:507] Master 127.25.14.61:37227 requested a full tablet report, sending...
I20260812 20:03:55.511885 17769 ts_manager.cc:194] Registered new tserver with Master: d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:55.511950 17683 ts_manager.cc:194] Registered new tserver with Master: d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:55.513099 17769 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.2:45301
W20260812 20:03:55.896147 18187 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 20:03:55.896587 18187 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 20:03:55.896742 18187 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 20:03:55.906092 18187 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 20:03:55.906348 18187 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.14.3
I20260812 20:03:55.917706 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.14.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.14.3
--webserver_port=0
--tserver_master_addrs=127.25.14.62:38865,127.25.14.61:37227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.14.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 565fa2a0c8b8655dfe08c1dca95e4b4d9408fc9c
build type FASTDEBUG
built by None at 12 Aug 2026 19:45:52 UTC on bdcb31816ec0
build id 13310
ASAN enabled
I20260812 20:03:55.919416 18187 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 20:03:55.921387 18187 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 20:03:55.932250 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:55.932282 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 20:03:55.932554 18197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 20:03:55.933954 18187 server_base.cc:1061] running on GCE node
I20260812 20:03:55.934545 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 20:03:55.935652 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 20:03:55.936939 18187 hybrid_clock.cc:648] HybridClock initialized: now 1786565035936892 us; error 56 us; skew 500 ppm
I20260812 20:03:55.937341 18187 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 20:03:55.939953 18187 webserver.cc:533] Webserver started at http://127.25.14.3:46205/ using document root <none> and password file <none>
I20260812 20:03:55.940869 18187 fs_manager.cc:362] Metadata directory not provided
I20260812 20:03:55.940984 18187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 20:03:55.941452 18187 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 20:03:55.944444 18187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data/instance:
uuid: "2aa0359db6f441889edca36cd80c4058"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.945482 18187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance:
uuid: "2aa0359db6f441889edca36cd80c4058"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.947911 18016 heartbeater.cc:499] Master 127.25.14.61:37227 was elected leader, sending a full tablet report...
I20260812 20:03:55.953109 18187 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260812 20:03:55.957387 18206 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:55.959164 18187 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 20:03:55.959424 18187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal
uuid: "2aa0359db6f441889edca36cd80c4058"
format_stamp: "Formatted at 2026-08-12 20:03:55 on dist-test-slave-p3t5"
I20260812 20:03:55.959692 18187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 20:03:55.977029 18187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260812 20:03:55.978056 18187 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 20:03:55.978387 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 20:03:55.979501 18187 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 20:03:55.981511 18187 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 20:03:55.981673 18187 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:55.981819 18187 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 20:03:55.981930 18187 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 20:03:56.023286 18187 rpc_server.cc:307] RPC server started. Bound to: 127.25.14.3:34495
I20260812 20:03:56.023340 18333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.14.3:34495 every 8 connection(s)
I20260812 20:03:56.024974 18187 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/data/info.pb
I20260812 20:03:56.027897 25656 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQcgiFK/build/asan/bin/kudu as pid 18187
I20260812 20:03:56.028115 25656 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/instance
I20260812 20:03:56.042971 18334 heartbeater.cc:344] Connected to a master server at 127.25.14.61:37227
I20260812 20:03:56.043139 18335 heartbeater.cc:344] Connected to a master server at 127.25.14.62:38865
I20260812 20:03:56.043459 18335 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:56.043464 18334 heartbeater.cc:461] Registering TS with master...
I20260812 20:03:56.044709 18334 heartbeater.cc:507] Master 127.25.14.61:37227 requested a full tablet report, sending...
I20260812 20:03:56.044755 18335 heartbeater.cc:507] Master 127.25.14.62:38865 requested a full tablet report, sending...
I20260812 20:03:56.046860 17769 ts_manager.cc:194] Registered new tserver with Master: 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.047127 17683 ts_manager.cc:194] Registered new tserver with Master: 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.047976 17769 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.14.3:42751
I20260812 20:03:56.053401 25656 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 20:03:56.081566 17769 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38702:
name: "table-0"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
W20260812 20:03:56.083197 17769 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 20:03:56.099373 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.099761 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.130498 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.018s sys 0.008s Metrics: {"bytes_written":694,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11901,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":615,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1286,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"thread_start_us":11756,"threads_started":1,"update_count":16}
I20260812 20:03:56.130805 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:56.131531 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.348219
I20260812 20:03:56.131673 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260812 20:03:56.131837 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.348219
I20260812 20:03:56.132205 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.138772 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.010s sys 0.019s Metrics: {"bytes_written":694,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11537,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1338,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1217,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"thread_start_us":11873,"threads_started":1,"update_count":16}
I20260812 20:03:56.139997 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.348219
I20260812 20:03:56.140363 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:56.140658 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.348219
I20260812 20:03:56.140991 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.348219
I20260812 20:03:56.146219 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20260812 20:03:56.146595 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.146872 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.176024 17938 tablet_service.cc:1511] Processing CreateTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.178485 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d7acaf0d9645ada0116c85aaac2c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.182471 17937 tablet_service.cc:1511] Processing CreateTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.183032 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdf5914e4334bfe9ea2ef52b14f0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.184511 17936 tablet_service.cc:1511] Processing CreateTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.184999 17936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637eaa46a444685a84db3a4a50c0e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.186251 17935 tablet_service.cc:1511] Processing CreateTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.186662 17935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f1ea51abb743b7b7b6b5feac8af23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.187875 17933 tablet_service.cc:1511] Processing CreateTablet for tablet 2f51859318ec4174be3dd534658149c2 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.188349 17933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f51859318ec4174be3dd534658149c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.189536 17932 tablet_service.cc:1511] Processing CreateTablet for tablet e3950e2b65124f5aa883e4e872b8777f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.190147 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3950e2b65124f5aa883e4e872b8777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.191579 17931 tablet_service.cc:1511] Processing CreateTablet for tablet 5a00feea243f440d9348756186c6df32 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.192154 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00feea243f440d9348756186c6df32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.193555 17929 tablet_service.cc:1511] Processing CreateTablet for tablet b8f649dddfa5466988e196d03656eb6f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.194123 17929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f649dddfa5466988e196d03656eb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.195523 17928 tablet_service.cc:1511] Processing CreateTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.196064 17928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfbe43f35904b94b26f8e6a239cf393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.197459 17927 tablet_service.cc:1511] Processing CreateTablet for tablet d50211ef1e684ab4824281af94b9febe (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.198122 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50211ef1e684ab4824281af94b9febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.201905 17926 tablet_service.cc:1511] Processing CreateTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.202497 17926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530fa3bb1bdf4fa083a35406188e2582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.204026 17925 tablet_service.cc:1511] Processing CreateTablet for tablet f7c30fa50d4049c2a3370881418edf80 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.204645 17925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c30fa50d4049c2a3370881418edf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.206293 17924 tablet_service.cc:1511] Processing CreateTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.206893 17924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fbf74c9ba7470ea946e2b437b3d71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.207307 18260 tablet_service.cc:1511] Processing CreateTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.208885 18257 tablet_service.cc:1511] Processing CreateTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.209580 18260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d7acaf0d9645ada0116c85aaac2c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.210155 17922 tablet_service.cc:1511] Processing CreateTablet for tablet f61d5f81507649418fcd65eee57dc3f3 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.210788 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61d5f81507649418fcd65eee57dc3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.211773 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdf5914e4334bfe9ea2ef52b14f0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.223050 17921 tablet_service.cc:1511] Processing CreateTablet for tablet ba44ce3aac624af697574a4274f38bf5 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.223161 17920 tablet_service.cc:1511] Processing CreateTablet for tablet c71fd265588a4fccae425a41ef98fb05 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.223742 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba44ce3aac624af697574a4274f38bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.224220 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71fd265588a4fccae425a41ef98fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.252692 18094 tablet_service.cc:1511] Processing CreateTablet for tablet d50211ef1e684ab4824281af94b9febe (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.253269 18101 tablet_service.cc:1511] Processing CreateTablet for tablet 2f51859318ec4174be3dd534658149c2 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.252290 18090 tablet_service.cc:1511] Processing CreateTablet for tablet f61d5f81507649418fcd65eee57dc3f3 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.254379 18093 tablet_service.cc:1511] Processing CreateTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.254825 18103 tablet_service.cc:1511] Processing CreateTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.254662 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.113s user 0.040s sys 0.045s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1674,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":686,"dirs.run_cpu_time_us":343,"dirs.run_wall_time_us":1328,"lbm_read_time_us":550,"lbm_reads_lt_1ms":21,"lbm_write_time_us":485,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":83352,"threads_started":1,"update_count":16}
I20260812 20:03:56.254977 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 5a00feea243f440d9348756186c6df32 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.252537 18096 tablet_service.cc:1511] Processing CreateTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.256132 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.124s user 0.027s sys 0.049s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1674,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":590,"dirs.run_cpu_time_us":307,"dirs.run_wall_time_us":1126,"lbm_read_time_us":415,"lbm_reads_lt_1ms":21,"lbm_write_time_us":268,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":107255,"threads_started":1,"update_count":16}
I20260812 20:03:56.260802 18087 tablet_service.cc:1511] Processing CreateTablet for tablet c71fd265588a4fccae425a41ef98fb05 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.261420 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50211ef1e684ab4824281af94b9febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.261862 18089 tablet_service.cc:1511] Processing CreateTablet for tablet ba44ce3aac624af697574a4274f38bf5 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.262867 18255 tablet_service.cc:1511] Processing CreateTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.263587 18255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f1ea51abb743b7b7b6b5feac8af23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.263945 18250 tablet_service.cc:1511] Processing CreateTablet for tablet 5a00feea243f440d9348756186c6df32 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.264535 18250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00feea243f440d9348756186c6df32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.266032 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.266321 18251 tablet_service.cc:1511] Processing CreateTablet for tablet e3950e2b65124f5aa883e4e872b8777f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.266532 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637eaa46a444685a84db3a4a50c0e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.267030 18251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3950e2b65124f5aa883e4e872b8777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.267416 18254 tablet_service.cc:1511] Processing CreateTablet for tablet 2f51859318ec4174be3dd534658149c2 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.267947 18254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f51859318ec4174be3dd534658149c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.268932 18245 tablet_service.cc:1511] Processing CreateTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.269484 18245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530fa3bb1bdf4fa083a35406188e2582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.270291 18247 tablet_service.cc:1511] Processing CreateTablet for tablet d50211ef1e684ab4824281af94b9febe (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.270933 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50211ef1e684ab4824281af94b9febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.274309 18249 tablet_service.cc:1511] Processing CreateTablet for tablet b8f649dddfa5466988e196d03656eb6f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.274935 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f649dddfa5466988e196d03656eb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.277757 18100 tablet_service.cc:1511] Processing CreateTablet for tablet e3950e2b65124f5aa883e4e872b8777f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.277889 18097 tablet_service.cc:1511] Processing CreateTablet for tablet b8f649dddfa5466988e196d03656eb6f (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.278276 18105 tablet_service.cc:1511] Processing CreateTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.278803 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.279268 18104 tablet_service.cc:1511] Processing CreateTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.279735 18104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637eaa46a444685a84db3a4a50c0e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.280989 18093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530fa3bb1bdf4fa083a35406188e2582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.281198 18092 tablet_service.cc:1511] Processing CreateTablet for tablet f7c30fa50d4049c2a3370881418edf80 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.281340 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71fd265588a4fccae425a41ef98fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.281678 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61d5f81507649418fcd65eee57dc3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.282254 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdf5914e4334bfe9ea2ef52b14f0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.283020 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba44ce3aac624af697574a4274f38bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.283618 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3950e2b65124f5aa883e4e872b8777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.285180 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfbe43f35904b94b26f8e6a239cf393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.285892 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fbf74c9ba7470ea946e2b437b3d71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.285826 18106 tablet_service.cc:1511] Processing CreateTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.286666 18241 tablet_service.cc:1511] Processing CreateTablet for tablet ba44ce3aac624af697574a4274f38bf5 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.287456 18241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba44ce3aac624af697574a4274f38bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.291898 18092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c30fa50d4049c2a3370881418edf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.292645 18103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f1ea51abb743b7b7b6b5feac8af23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.293953 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00feea243f440d9348756186c6df32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.294267 18101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f51859318ec4174be3dd534658149c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.329929 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f649dddfa5466988e196d03656eb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.342072 18106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d7acaf0d9645ada0116c85aaac2c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.360419 18375 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.369248 18375 tablet_bootstrap.cc:654] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.371690 18375 log.cc:826] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:56.390774 18242 tablet_service.cc:1511] Processing CreateTablet for tablet f61d5f81507649418fcd65eee57dc3f3 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.392570 18244 tablet_service.cc:1511] Processing CreateTablet for tablet f7c30fa50d4049c2a3370881418edf80 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.393275 18248 tablet_service.cc:1511] Processing CreateTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.393941 18240 tablet_service.cc:1511] Processing CreateTablet for tablet c71fd265588a4fccae425a41ef98fb05 (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.396064 18248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfbe43f35904b94b26f8e6a239cf393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.396878 18244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c30fa50d4049c2a3370881418edf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.397011 18243 tablet_service.cc:1511] Processing CreateTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e (DEFAULT_TABLE table=table-0 [id=fdf0b43d520b4768a902707b8b2719bc]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:56.400089 18375 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.400642 18375 ts_tablet_manager.cc:1403] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.007s
I20260812 20:03:56.420267 18374 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.422417 18375 raft_consensus.cc:359] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.423030 18375 raft_consensus.cc:385] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.423188 18375 raft_consensus.cc:740] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.424022 18375 consensus_queue.cc:260] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.424815 18374 tablet_bootstrap.cc:654] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.426481 18374 log.cc:826] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:56.446026 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71fd265588a4fccae425a41ef98fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.461021 18243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fbf74c9ba7470ea946e2b437b3d71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.492208 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61d5f81507649418fcd65eee57dc3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:56.499331 18373 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.502633 18374 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.502959 18374 ts_tablet_manager.cc:1403] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.083s user 0.037s sys 0.044s
I20260812 20:03:56.505373 18178 heartbeater.cc:499] Master 127.25.14.61:37227 was elected leader, sending a full tablet report...
I20260812 20:03:56.509356 18375 ts_tablet_manager.cc:1434] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.108s user 0.051s sys 0.039s
I20260812 20:03:56.509874 18375 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.510888 18374 raft_consensus.cc:359] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.511373 18374 raft_consensus.cc:385] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.511478 18374 raft_consensus.cc:740] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.511955 18375 tablet_bootstrap.cc:654] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.512351 18374 consensus_queue.cc:260] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.526364 18373 tablet_bootstrap.cc:654] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.528844 18373 log.cc:826] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Log is configured to *not* fsync() on all Append() calls
I20260812 20:03:56.562938 18374 ts_tablet_manager.cc:1434] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.060s user 0.015s sys 0.024s
I20260812 20:03:56.566136 18374 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.568277 18374 tablet_bootstrap.cc:654] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.568699 18379 raft_consensus.cc:493] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.570293 18379 raft_consensus.cc:515] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.574698 18379 leader_election.cc:290] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.575466 18373 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.575805 18373 ts_tablet_manager.cc:1403] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.104s user 0.013s sys 0.004s
I20260812 20:03:56.576049 18375 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.576303 18375 ts_tablet_manager.cc:1403] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.067s user 0.027s sys 0.020s
I20260812 20:03:56.578212 18375 raft_consensus.cc:359] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.578851 18375 raft_consensus.cc:385] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.579147 18375 raft_consensus.cc:740] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.579568 18375 consensus_queue.cc:260] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.580587 18375 ts_tablet_manager.cc:1434] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 20:03:56.583271 18374 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.583582 18374 ts_tablet_manager.cc:1403] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260812 20:03:56.584458 18373 raft_consensus.cc:359] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.584910 18373 raft_consensus.cc:385] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.585042 18373 raft_consensus.cc:740] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.585314 18374 raft_consensus.cc:359] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.585696 18374 raft_consensus.cc:385] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.585891 18374 raft_consensus.cc:740] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.586028 18373 consensus_queue.cc:260] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.586495 18374 consensus_queue.cc:260] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.587477 18374 ts_tablet_manager.cc:1434] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260812 20:03:56.588151 18374 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.591157 18374 tablet_bootstrap.cc:654] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.593783 18334 heartbeater.cc:499] Master 127.25.14.61:37227 was elected leader, sending a full tablet report...
I20260812 20:03:56.594276 18375 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.596221 18375 tablet_bootstrap.cc:654] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.601923 18386 raft_consensus.cc:493] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.602201 18386 raft_consensus.cc:515] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.602555 18375 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.602763 18375 ts_tablet_manager.cc:1403] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260812 20:03:56.603642 18375 raft_consensus.cc:359] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.603845 18375 raft_consensus.cc:385] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.603914 18375 raft_consensus.cc:740] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.604393 18375 consensus_queue.cc:260] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.604940 18375 ts_tablet_manager.cc:1434] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:56.605350 18375 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.607359 18375 tablet_bootstrap.cc:654] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.609539 18374 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.609755 18374 ts_tablet_manager.cc:1403] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.007s
I20260812 20:03:56.610772 18374 raft_consensus.cc:359] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.611039 18374 raft_consensus.cc:385] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.611129 18374 raft_consensus.cc:740] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.611256 18373 ts_tablet_manager.cc:1434] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.035s user 0.015s sys 0.004s
I20260812 20:03:56.611441 18374 consensus_queue.cc:260] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.612085 18374 ts_tablet_manager.cc:1434] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:56.612509 18374 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.615167 18374 tablet_bootstrap.cc:654] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.620071 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:56.623852 17883 leader_election.cc:343] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.636404 18386 leader_election.cc:290] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:56.639070 18373 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.641502 18373 tablet_bootstrap.cc:654] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.642237 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.643545 18374 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.643870 18374 ts_tablet_manager.cc:1403] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.028s
W20260812 20:03:56.644176 17884 leader_election.cc:343] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.644389 17884 leader_election.cc:304] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:56.644996 18379 raft_consensus.cc:2749] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:56.645658 18374 raft_consensus.cc:359] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.645864 18374 raft_consensus.cc:385] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.645992 18374 raft_consensus.cc:740] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.646380 18374 consensus_queue.cc:260] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.647035 18374 ts_tablet_manager.cc:1434] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 20:03:56.647430 18374 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.649739 18374 tablet_bootstrap.cc:654] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.649735 18375 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.649919 18375 ts_tablet_manager.cc:1403] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.016s
I20260812 20:03:56.650875 18375 raft_consensus.cc:359] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.651075 18375 raft_consensus.cc:385] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.651160 18375 raft_consensus.cc:740] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.651396 18375 consensus_queue.cc:260] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.652619 18375 ts_tablet_manager.cc:1434] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:56.653024 18375 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.653772 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:56.655129 18375 tablet_bootstrap.cc:654] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:56.655565 18207 leader_election.cc:343] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.662060 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:56.663014 18208 leader_election.cc:343] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.663328 18208 leader_election.cc:304] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:56.664013 18386 raft_consensus.cc:2749] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:56.666906 18375 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.666929 18373 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.667079 18374 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.667089 18373 ts_tablet_manager.cc:1403] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20260812 20:03:56.667114 18375 ts_tablet_manager.cc:1403] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260812 20:03:56.667276 18374 ts_tablet_manager.cc:1403] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.002s
I20260812 20:03:56.667937 18375 raft_consensus.cc:359] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.667972 18374 raft_consensus.cc:359] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.668177 18375 raft_consensus.cc:385] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.668226 18374 raft_consensus.cc:385] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.668260 18375 raft_consensus.cc:740] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.668188 18373 raft_consensus.cc:359] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.668310 18374 raft_consensus.cc:740] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.668377 18373 raft_consensus.cc:385] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.668484 18373 raft_consensus.cc:740] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.668610 18375 consensus_queue.cc:260] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.668675 18374 consensus_queue.cc:260] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.668722 18373 consensus_queue.cc:260] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.669189 18375 ts_tablet_manager.cc:1434] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 20:03:56.669411 18373 ts_tablet_manager.cc:1434] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 20:03:56.669549 18375 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.669772 18373 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.670936 18374 ts_tablet_manager.cc:1434] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 20:03:56.671274 18374 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.671532 18375 tablet_bootstrap.cc:654] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.671552 18373 tablet_bootstrap.cc:654] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.674170 18374 tablet_bootstrap.cc:654] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:56.674382 18019 tablet.cc:2406] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:03:56.686694 18374 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.686921 18374 ts_tablet_manager.cc:1403] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20260812 20:03:56.687856 18374 raft_consensus.cc:359] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.688087 18374 raft_consensus.cc:385] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.688195 18374 raft_consensus.cc:740] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.688512 18374 consensus_queue.cc:260] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.689086 18374 ts_tablet_manager.cc:1434] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:56.689442 18374 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.690985 18377 raft_consensus.cc:493] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.691452 18374 tablet_bootstrap.cc:654] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.691329 18377 raft_consensus.cc:515] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.694664 18377 leader_election.cc:290] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.695925 18374 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.696121 18374 ts_tablet_manager.cc:1403] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 20:03:56.697080 18374 raft_consensus.cc:359] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.697338 18374 raft_consensus.cc:385] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.697422 18374 raft_consensus.cc:740] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.697816 18374 consensus_queue.cc:260] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.700273 18374 ts_tablet_manager.cc:1434] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 20:03:56.700657 18374 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.702723 18374 tablet_bootstrap.cc:654] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.708680 18373 tablet_bootstrap.cc:492] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.708951 18373 ts_tablet_manager.cc:1403] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.019s
I20260812 20:03:56.710177 18373 raft_consensus.cc:359] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.710480 18373 raft_consensus.cc:385] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.710567 18373 raft_consensus.cc:740] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.710899 18373 consensus_queue.cc:260] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.711675 18373 ts_tablet_manager.cc:1434] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:56.712141 18373 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.715147 18375 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.715382 18375 ts_tablet_manager.cc:1403] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.014s
I20260812 20:03:56.716562 18375 raft_consensus.cc:359] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.716835 18375 raft_consensus.cc:385] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.716958 18375 raft_consensus.cc:740] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.717365 18375 consensus_queue.cc:260] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.718101 18375 ts_tablet_manager.cc:1434] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:56.718530 18375 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.721246 18375 tablet_bootstrap.cc:654] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.722849 18373 tablet_bootstrap.cc:654] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.725253 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "530fa3bb1bdf4fa083a35406188e2582" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:56.725664 17964 raft_consensus.cc:2468] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:56.731937 18379 raft_consensus.cc:493] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.732156 18379 raft_consensus.cc:515] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.733022 18379 leader_election.cc:290] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.733667 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f1ea51abb743b7b7b6b5feac8af23b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:56.734411 17883 leader_election.cc:343] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.735416 18043 leader_election.cc:304] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.736114 18377 raft_consensus.cc:2804] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.736304 18377 raft_consensus.cc:493] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.736420 18377 raft_consensus.cc:3060] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.740098 18377 raft_consensus.cc:515] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.741066 18377 leader_election.cc:290] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
W20260812 20:03:56.742187 18181 tablet.cc:2406] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:03:56.743285 18377 raft_consensus.cc:493] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.743522 18377 raft_consensus.cc:515] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.744463 18377 leader_election.cc:290] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.745571 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:56.745797 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f1ea51abb743b7b7b6b5feac8af23b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.745976 17962 raft_consensus.cc:2468] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:56.746026 18127 raft_consensus.cc:2468] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:56.746345 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "530fa3bb1bdf4fa083a35406188e2582" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:56.746570 17964 raft_consensus.cc:3060] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.747188 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "530fa3bb1bdf4fa083a35406188e2582" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.747421 18281 raft_consensus.cc:2468] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:56.748147 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "530fa3bb1bdf4fa083a35406188e2582" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:56.748345 18281 raft_consensus.cc:3060] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.750170 17964 raft_consensus.cc:2468] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:56.750803 17884 leader_election.cc:304] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:56.751377 18379 raft_consensus.cc:2804] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.751523 18379 raft_consensus.cc:493] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.751633 18379 raft_consensus.cc:3060] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.751791 18281 raft_consensus.cc:2468] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:56.753008 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.753666 18043 leader_election.cc:304] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.754065 18043 leader_election.cc:304] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
W20260812 20:03:56.754896 18043 leader_election.cc:343] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.755091 18379 raft_consensus.cc:515] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.755364 18377 raft_consensus.cc:2804] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.755486 18377 raft_consensus.cc:493] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.755574 18377 raft_consensus.cc:3060] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.755919 18379 leader_election.cc:290] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.756942 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f1ea51abb743b7b7b6b5feac8af23b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
W20260812 20:03:56.757586 17883 leader_election.cc:343] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.758002 18406 raft_consensus.cc:2804] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.758665 18377 raft_consensus.cc:515] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.759727 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f1ea51abb743b7b7b6b5feac8af23b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:56.759989 18127 raft_consensus.cc:3060] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.764631 18127 raft_consensus.cc:2468] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:56.765291 18406 raft_consensus.cc:697] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:56.765437 17884 leader_election.cc:304] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:56.766137 18379 raft_consensus.cc:2804] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.766325 18379 raft_consensus.cc:697] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:56.766158 18406 consensus_queue.cc:237] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.766904 18379 consensus_queue.cc:237] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.776978 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:56.777231 17964 raft_consensus.cc:3060] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:03:56.777948 18336 tablet.cc:2406] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 20:03:56.780375 17964 raft_consensus.cc:2468] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:56.782159 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:56.783176 17768 catalog_manager.cc:5719] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:56.784957 18377 leader_election.cc:290] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.793229 18043 leader_election.cc:304] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
W20260812 20:03:56.793975 18043 leader_election.cc:343] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.794678 18377 raft_consensus.cc:2804] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.795042 18377 raft_consensus.cc:697] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:56.795444 18377 consensus_queue.cc:237] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.798908 18374 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.799137 18374 ts_tablet_manager.cc:1403] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.099s user 0.038s sys 0.049s
I20260812 20:03:56.800320 18374 raft_consensus.cc:359] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.800590 18374 raft_consensus.cc:385] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.800706 18374 raft_consensus.cc:740] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.805301 18374 consensus_queue.cc:260] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.805513 18373 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.805709 18373 ts_tablet_manager.cc:1403] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.094s user 0.024s sys 0.018s
I20260812 20:03:56.806087 18374 ts_tablet_manager.cc:1434] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260812 20:03:56.806423 18374 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.806644 18373 raft_consensus.cc:359] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.806872 18373 raft_consensus.cc:385] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.806978 18373 raft_consensus.cc:740] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.807302 18373 consensus_queue.cc:260] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.809149 18373 ts_tablet_manager.cc:1434] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:56.809597 18373 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.810667 18374 tablet_bootstrap.cc:654] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.811609 18373 tablet_bootstrap.cc:654] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.816675 18379 raft_consensus.cc:493] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.816931 18379 raft_consensus.cc:515] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.818593 18375 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.818850 18375 ts_tablet_manager.cc:1403] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.031s
I20260812 20:03:56.819548 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7c30fa50d4049c2a3370881418edf80" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.819908 18127 raft_consensus.cc:2468] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:56.819964 18375 raft_consensus.cc:359] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.820132 18374 tablet_bootstrap.cc:492] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.820187 18375 raft_consensus.cc:385] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.820266 18375 raft_consensus.cc:740] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.820331 18374 ts_tablet_manager.cc:1403] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20260812 20:03:56.820573 18375 consensus_queue.cc:260] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.820997 17884 leader_election.cc:304] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.821135 18375 ts_tablet_manager.cc:1434] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 20:03:56.821170 18374 raft_consensus.cc:359] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.821393 18374 raft_consensus.cc:385] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.821473 18374 raft_consensus.cc:740] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.821718 18381 raft_consensus.cc:2804] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.821848 18381 raft_consensus.cc:493] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.821987 18375 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.821995 18381 raft_consensus.cc:3060] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.823881 18374 consensus_queue.cc:260] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:56.824503 18375 tablet_bootstrap.cc:654] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.826958 18381 raft_consensus.cc:515] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.828045 18373 tablet_bootstrap.cc:492] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.828271 18373 ts_tablet_manager.cc:1403] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20260812 20:03:56.828944 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7c30fa50d4049c2a3370881418edf80" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:56.829246 18127 raft_consensus.cc:3060] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.829380 18373 raft_consensus.cc:359] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.829666 18373 raft_consensus.cc:385] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.829767 18373 raft_consensus.cc:740] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.830090 18373 consensus_queue.cc:260] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.831339 18373 ts_tablet_manager.cc:1434] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:03:56.831729 18373 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.832324 18374 ts_tablet_manager.cc:1434] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20260812 20:03:56.832741 18374 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.835114 18373 tablet_bootstrap.cc:654] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.836796 18375 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.836984 18375 ts_tablet_manager.cc:1403] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260812 20:03:56.837103 18127 raft_consensus.cc:2468] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:56.837754 18374 tablet_bootstrap.cc:654] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.838006 17884 leader_election.cc:304] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.837976 18375 raft_consensus.cc:359] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.838217 18375 raft_consensus.cc:385] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.838320 18375 raft_consensus.cc:740] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.838649 18375 consensus_queue.cc:260] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.838882 18418 raft_consensus.cc:2804] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.839432 18375 ts_tablet_manager.cc:1434] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:56.839795 18375 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.840214 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7c30fa50d4049c2a3370881418edf80" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:56.840468 18281 raft_consensus.cc:3060] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.842214 18375 tablet_bootstrap.cc:654] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.842657 18418 raft_consensus.cc:697] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:56.843026 18418 consensus_queue.cc:237] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.844903 18281 raft_consensus.cc:2468] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:56.847409 18381 leader_election.cc:290] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.849270 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7c30fa50d4049c2a3370881418edf80" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.849790 18281 raft_consensus.cc:2376] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1. Re-sending same reply.
I20260812 20:03:56.850270 18381 raft_consensus.cc:493] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.850621 18381 raft_consensus.cc:515] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.851064 18379 leader_election.cc:290] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.851660 18381 leader_election.cc:290] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.852552 18374 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.853121 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.853521 18374 ts_tablet_manager.cc:1403] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
W20260812 20:03:56.853986 17883 leader_election.cc:343] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.854594 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.855000 18127 raft_consensus.cc:2468] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:56.855299 18374 raft_consensus.cc:359] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.855580 18374 raft_consensus.cc:385] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.855705 18374 raft_consensus.cc:740] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.855772 17884 leader_election.cc:304] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:56.856716 18381 raft_consensus.cc:2804] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.856906 18381 raft_consensus.cc:493] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.857121 18381 raft_consensus.cc:3060] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.857417 18386 raft_consensus.cc:493] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.857640 18386 raft_consensus.cc:515] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.858632 18386 leader_election.cc:290] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:56.860143 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.860215 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:03:56.860888 18208 leader_election.cc:343] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:56.861691 18207 leader_election.cc:343] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.861860 18207 leader_election.cc:304] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:56.862181 18375 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.862387 18375 ts_tablet_manager.cc:1403] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20260812 20:03:56.862433 18386 raft_consensus.cc:2749] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:56.863493 18375 raft_consensus.cc:359] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.863765 18375 raft_consensus.cc:385] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.863868 18375 raft_consensus.cc:740] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.860088 18374 consensus_queue.cc:260] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.864274 18375 consensus_queue.cc:260] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.865141 18374 ts_tablet_manager.cc:1434] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20260812 20:03:56.865075 18375 ts_tablet_manager.cc:1434] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:56.865319 18418 raft_consensus.cc:493] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.866466 18373 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.866338 18418 raft_consensus.cc:515] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.866652 18373 ts_tablet_manager.cc:1403] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20260812 20:03:56.867029 18381 raft_consensus.cc:515] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.867679 18373 raft_consensus.cc:359] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.867889 18373 raft_consensus.cc:385] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.868007 18373 raft_consensus.cc:740] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.868328 18373 consensus_queue.cc:260] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.868891 18375 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.869089 18373 ts_tablet_manager.cc:1434] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:56.869412 18373 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.870160 17769 catalog_manager.cc:5719] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:56.871580 18373 tablet_bootstrap.cc:654] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.871682 18375 tablet_bootstrap.cc:654] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.873273 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71fd265588a4fccae425a41ef98fb05" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.873692 18127 raft_consensus.cc:2468] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:56.879875 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71fd265588a4fccae425a41ef98fb05" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.868114 18418 leader_election.cc:290] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.868716 18374 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.874526 17884 leader_election.cc:304] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.881394 18418 raft_consensus.cc:2804] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.881515 18418 raft_consensus.cc:493] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.881676 18418 raft_consensus.cc:3060] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.883359 18374 tablet_bootstrap.cc:654] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.886009 18418 raft_consensus.cc:515] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.886214 18375 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.886399 18375 ts_tablet_manager.cc:1403] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260812 20:03:56.887351 18375 raft_consensus.cc:359] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.887573 18375 raft_consensus.cc:385] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.887693 18375 raft_consensus.cc:740] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.887962 18375 consensus_queue.cc:260] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.888538 18375 ts_tablet_manager.cc:1434] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:56.888875 18375 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.890457 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:56.890803 18375 tablet_bootstrap.cc:654] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.892918 18373 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.893107 18373 ts_tablet_manager.cc:1403] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.003s
I20260812 20:03:56.894129 18373 raft_consensus.cc:359] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.894332 18373 raft_consensus.cc:385] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.894409 18373 raft_consensus.cc:740] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.894675 18373 consensus_queue.cc:260] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.895256 18373 ts_tablet_manager.cc:1434] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 20:03:56.895505 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.895619 18373 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.895913 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.896235 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.896589 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.897035 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.897334 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.897462 18373 tablet_bootstrap.cc:654] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.897493 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.900415 18375 tablet_bootstrap.cc:492] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.900612 18375 ts_tablet_manager.cc:1403] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260812 20:03:56.901502 18375 raft_consensus.cc:359] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.901788 18375 raft_consensus.cc:385] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.901865 18375 raft_consensus.cc:740] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.902918 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":220,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:03:56.904402 18375 consensus_queue.cc:260] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.905105 18375 ts_tablet_manager.cc:1434] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 20:03:56.905483 18375 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.905681 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:56.905871 18373 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.906044 18373 ts_tablet_manager.cc:1403] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260812 20:03:56.906455 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.905975 18127 raft_consensus.cc:3060] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.907061 18373 raft_consensus.cc:359] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.886915 18381 leader_election.cc:290] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.907626 18375 tablet_bootstrap.cc:654] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.907578 18381 raft_consensus.cc:493] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.907778 18381 raft_consensus.cc:515] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.908613 18381 leader_election.cc:290] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.888078 18374 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
W20260812 20:03:56.889910 17883 leader_election.cc:343] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.908988 18373 raft_consensus.cc:385] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.909096 18373 raft_consensus.cc:740] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.909375 18373 consensus_queue.cc:260] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:56.909607 17883 leader_election.cc:343] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.908902 18374 ts_tablet_manager.cc:1403] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
I20260812 20:03:56.910569 18374 raft_consensus.cc:359] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.910778 18374 raft_consensus.cc:385] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.910848 18374 raft_consensus.cc:740] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.911094 18374 consensus_queue.cc:260] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.911631 18374 ts_tablet_manager.cc:1434] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:56.911960 18374 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.912721 18373 ts_tablet_manager.cc:1434] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260812 20:03:56.913259 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.001s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":189,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:03:56.915218 18373 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.916229 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50211ef1e684ab4824281af94b9febe" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:56.916476 18281 raft_consensus.cc:2468] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:56.916682 18418 leader_election.cc:290] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.918615 18127 raft_consensus.cc:2468] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:56.919819 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50211ef1e684ab4824281af94b9febe" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.920567 17884 leader_election.cc:304] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:56.920912 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71fd265588a4fccae425a41ef98fb05" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:56.921132 18126 raft_consensus.cc:3060] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:03:56.921119 17884 leader_election.cc:343] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.922159 18418 raft_consensus.cc:2804] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.922497 18418 raft_consensus.cc:697] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:56.922864 18418 consensus_queue.cc:237] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.924337 18126 raft_consensus.cc:2468] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:56.927012 17883 leader_election.cc:304] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:56.927793 18381 raft_consensus.cc:2804] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.927932 18381 raft_consensus.cc:493] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.928069 18381 raft_consensus.cc:3060] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.929739 18373 tablet_bootstrap.cc:654] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.933116 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71fd265588a4fccae425a41ef98fb05" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
W20260812 20:03:56.933892 17883 leader_election.cc:343] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.934834 17884 leader_election.cc:304] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:56.935724 18374 tablet_bootstrap.cc:654] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.938024 18418 raft_consensus.cc:2804] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:56.938167 17768 catalog_manager.cc:5719] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:56.938715 18418 raft_consensus.cc:697] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:56.939107 18418 consensus_queue.cc:237] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.932426 18381 raft_consensus.cc:515] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.942605 18381 leader_election.cc:290] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.944651 18374 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:56.944871 18374 ts_tablet_manager.cc:1403] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20260812 20:03:56.945976 18374 raft_consensus.cc:359] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.946259 18374 raft_consensus.cc:385] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.946347 18374 raft_consensus.cc:740] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.946614 18374 consensus_queue.cc:260] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.947540 18374 ts_tablet_manager.cc:1434] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:56.949977 18373 tablet_bootstrap.cc:492] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:56.950243 18373 ts_tablet_manager.cc:1403] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.004s
I20260812 20:03:56.950629 18374 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:56.951581 18373 raft_consensus.cc:359] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.952297 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50211ef1e684ab4824281af94b9febe" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:56.952601 18373 raft_consensus.cc:385] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.952749 18373 raft_consensus.cc:740] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
W20260812 20:03:56.952986 17884 leader_election.cc:343] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.953190 18373 consensus_queue.cc:260] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.953449 18374 tablet_bootstrap.cc:654] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.953926 18373 ts_tablet_manager.cc:1434] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:56.954397 18373 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:56.954478 17769 catalog_manager.cc:5719] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:56.955211 18375 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:56.955376 18375 ts_tablet_manager.cc:1403] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.009s
I20260812 20:03:56.956197 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.956264 18373 tablet_bootstrap.cc:654] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.956247 18375 raft_consensus.cc:359] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.956523 18375 raft_consensus.cc:385] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:56.956622 18375 raft_consensus.cc:740] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:56.956616 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.956876 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.956943 18375 consensus_queue.cc:260] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.957090 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.957234 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.957520 18375 ts_tablet_manager.cc:1434] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:56.957562 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.957908 18375 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:56.957891 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.958527 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.958869 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.959123 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.959786 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.959825 18375 tablet_bootstrap.cc:654] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:56.960122 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.960502 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.960924 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.961225 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.962368 18377 raft_consensus.cc:493] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.962551 18377 raft_consensus.cc:515] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.963202 18377 leader_election.cc:290] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:56.963416 18386 raft_consensus.cc:493] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:56.963622 18386 raft_consensus.cc:515] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.964495 18386 leader_election.cc:290] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:56.966224 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637eaa46a444685a84db3a4a50c0e3b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:56.966508 17964 raft_consensus.cc:2468] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:56.967412 18207 leader_election.cc:304] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:56.967883 18386 raft_consensus.cc:2804] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:56.968029 18386 raft_consensus.cc:493] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:56.968139 18386 raft_consensus.cc:3060] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.971311 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.971740 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.972059 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.217792
I20260812 20:03:56.972321 18386 raft_consensus.cc:515] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:56.972505 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.972808 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.973151 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"mutex_wait_us":589,"reinsert_count":0,"update_count":1}
I20260812 20:03:56.973974 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:56.974265 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:56.974164 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637eaa46a444685a84db3a4a50c0e3b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:56.974622 17964 raft_consensus.cc:3060] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:56.985996 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.986765 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.987784 17964 raft_consensus.cc:2468] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:56.988269 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637eaa46a444685a84db3a4a50c0e3b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:56.988880 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.989279 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.989591 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:56.990944 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:56.992355 18043 leader_election.cc:343] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.990355 18126 raft_consensus.cc:2468] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:56.990108 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:56.994540 18207 leader_election.cc:304] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
W20260812 20:03:56.996865 18043 leader_election.cc:343] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:56.997857 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.041s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":201,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":7168,"update_count":1}
I20260812 20:03:56.998412 18043 leader_election.cc:304] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:56.999619 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637eaa46a444685a84db3a4a50c0e3b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.000469 18126 raft_consensus.cc:3060] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.004691 18386 leader_election.cc:290] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.005178 18377 raft_consensus.cc:2749] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.007115 18377 raft_consensus.cc:493] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.007334 18377 raft_consensus.cc:515] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.007754 18418 raft_consensus.cc:493] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.007966 18418 raft_consensus.cc:515] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:57.008168 18377 leader_election.cc:290] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.008667 18418 leader_election.cc:290] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.005578 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50211ef1e684ab4824281af94b9febe" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.008989 18281 raft_consensus.cc:3060] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.011288 17767 catalog_manager.cc:5719] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.013434 18281 raft_consensus.cc:2468] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.014302 18386 raft_consensus.cc:2804] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.015797 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.016228 17964 raft_consensus.cc:2468] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:57.016956 18386 raft_consensus.cc:697] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:57.018307 18386 consensus_queue.cc:237] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.020406 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.020690 18127 raft_consensus.cc:2468] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.021008 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.021417 17884 leader_election.cc:304] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.021911 18418 raft_consensus.cc:2804] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.022020 18418 raft_consensus.cc:493] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.022142 18418 raft_consensus.cc:3060] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.024070 17883 leader_election.cc:304] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:57.024447 18381 raft_consensus.cc:2804] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.024644 18381 raft_consensus.cc:697] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.024948 18381 consensus_queue.cc:237] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.026413 18418 raft_consensus.cc:515] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:57.027597 18418 leader_election.cc:290] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.012602 17767 catalog_manager.cc:5719] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.019436 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.029937 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.039482 18279 raft_consensus.cc:2468] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.033639 18278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.039881 18278 raft_consensus.cc:3060] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.040004 18443 raft_consensus.cc:493] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.040196 18443 raft_consensus.cc:515] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.041975 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.044127 18278 raft_consensus.cc:2468] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.044658 18443 leader_election.cc:290] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.047087 17883 leader_election.cc:304] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.047667 18418 raft_consensus.cc:2804] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.048025 18418 raft_consensus.cc:697] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.048406 18418 consensus_queue.cc:237] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:57.050256 18043 leader_election.cc:304] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.050719 18377 raft_consensus.cc:2804] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.050843 18377 raft_consensus.cc:493] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.050979 18377 raft_consensus.cc:3060] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.054963 18377 raft_consensus.cc:515] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.055897 18377 leader_election.cc:290] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.056830 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:57.057113 17964 raft_consensus.cc:3060] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.061327 17964 raft_consensus.cc:2468] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
W20260812 20:03:57.062597 18043 leader_election.cc:343] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.063459 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11d7acaf0d9645ada0116c85aaac2c40" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.063737 18127 raft_consensus.cc:3060] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.067927 18127 raft_consensus.cc:2468] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.069330 18043 leader_election.cc:304] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.069896 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.070593 18443 raft_consensus.cc:493] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.070802 18443 raft_consensus.cc:515] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.071444 18443 leader_election.cc:290] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.072278 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:57.072937 18208 leader_election.cc:343] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.072965 18126 raft_consensus.cc:2468] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:57.075277 18377 raft_consensus.cc:2804] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.076175 18377 raft_consensus.cc:697] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:57.076674 18377 consensus_queue.cc:237] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.078889 17767 catalog_manager.cc:5719] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.079550 17767 catalog_manager.cc:5719] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.092079 17768 catalog_manager.cc:5719] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
W20260812 20:03:57.092561 18043 leader_election.cc:343] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.098783 18377 raft_consensus.cc:493] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.099018 18377 raft_consensus.cc:515] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.099987 18377 leader_election.cc:290] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.100886 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.102114 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:57.102926 18043 leader_election.cc:343] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:57.103197 18043 leader_election.cc:343] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.103323 18043 leader_election.cc:304] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.104128 18377 raft_consensus.cc:2749] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 20:03:57.107185 18207 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.14.1:46337, user_credentials={real_user=slave}} blocked reactor thread for 112683us
I20260812 20:03:57.108124 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f51859318ec4174be3dd534658149c2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.108585 17964 raft_consensus.cc:2393] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate d222b63b24184ca6bc530a1547fe3297 in this term.
I20260812 20:03:57.111829 17767 catalog_manager.cc:5719] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:57.112282 18374 tablet_bootstrap.cc:492] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.112485 18374 ts_tablet_manager.cc:1403] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.165s user 0.026s sys 0.052s
I20260812 20:03:57.113415 18374 raft_consensus.cc:359] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.113674 18374 raft_consensus.cc:385] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.113821 18374 raft_consensus.cc:740] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.114123 18374 consensus_queue.cc:260] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.109691 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.115414 18374 ts_tablet_manager.cc:1434] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:03:57.115343 18207 leader_election.cc:304] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.115754 18374 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
W20260812 20:03:57.115936 18207 leader_election.cc:343] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.116092 18207 leader_election.cc:304] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.116346 18386 raft_consensus.cc:3060] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.117830 18374 tablet_bootstrap.cc:654] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.120196 18386 raft_consensus.cc:2749] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.120537 18386 raft_consensus.cc:2749] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.121229 18374 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.121394 18374 ts_tablet_manager.cc:1403] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 20:03:57.122339 18374 raft_consensus.cc:359] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.122545 18374 raft_consensus.cc:385] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.122637 18374 raft_consensus.cc:740] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.122848 18374 consensus_queue.cc:260] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.123425 18374 ts_tablet_manager.cc:1434] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:57.123728 18374 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:57.125495 18374 tablet_bootstrap.cc:654] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.129484 18374 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.129709 18374 ts_tablet_manager.cc:1403] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260812 20:03:57.130529 18374 raft_consensus.cc:359] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.130738 18374 raft_consensus.cc:385] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.130812 18374 raft_consensus.cc:740] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.131156 18374 consensus_queue.cc:260] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.131788 18374 ts_tablet_manager.cc:1434] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:57.132138 18374 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:57.133985 18374 tablet_bootstrap.cc:654] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.138342 18374 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.138505 18374 ts_tablet_manager.cc:1403] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 20:03:57.139313 18374 raft_consensus.cc:359] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.139505 18374 raft_consensus.cc:385] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.139624 18374 raft_consensus.cc:740] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.139890 18374 consensus_queue.cc:260] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.140584 18374 ts_tablet_manager.cc:1434] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:57.142005 18418 raft_consensus.cc:493] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.142181 18418 raft_consensus.cc:515] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.142845 18418 leader_election.cc:290] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.143932 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.144217 18126 raft_consensus.cc:2468] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.144682 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.145567 17884 leader_election.cc:304] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.146019 18418 raft_consensus.cc:2804] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.146210 18418 raft_consensus.cc:493] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.146330 18418 raft_consensus.cc:3060] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.150138 18418 raft_consensus.cc:515] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.151105 18375 tablet_bootstrap.cc:492] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:57.151224 18381 raft_consensus.cc:493] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.151319 18375 ts_tablet_manager.cc:1403] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.193s user 0.021s sys 0.054s
I20260812 20:03:57.151439 18381 raft_consensus.cc:515] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.151504 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.151712 18126 raft_consensus.cc:3060] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.152315 18381 leader_election.cc:290] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.152349 18375 raft_consensus.cc:359] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.152580 18375 raft_consensus.cc:385] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.152667 18375 raft_consensus.cc:740] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.152948 18375 consensus_queue.cc:260] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.153539 18375 ts_tablet_manager.cc:1434] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:57.153985 18375 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:57.156069 18375 tablet_bootstrap.cc:654] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.156214 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:57.157470 17884 leader_election.cc:343] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.159998 18375 tablet_bootstrap.cc:492] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:57.160269 18375 ts_tablet_manager.cc:1403] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260812 20:03:57.160279 18126 raft_consensus.cc:2468] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.160961 17884 leader_election.cc:304] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.161345 18381 raft_consensus.cc:2804] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.161522 18381 raft_consensus.cc:697] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.161667 18375 raft_consensus.cc:359] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.161835 18381 consensus_queue.cc:237] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.161966 18375 raft_consensus.cc:385] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.162186 18375 raft_consensus.cc:740] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.162513 18375 consensus_queue.cc:260] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.163239 18375 ts_tablet_manager.cc:1434] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 20:03:57.163672 18375 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:57.163532 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.176s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":176328}
I20260812 20:03:57.163671 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.128s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":128374}
I20260812 20:03:57.164475 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.178s user 0.005s sys 0.001s Metrics: {"bytes_written":609,"delete_count":0,"lbm_write_time_us":210,"lbm_writes_lt_1ms":4,"mutex_wait_us":171619,"reinsert_count":0,"update_count":7}
I20260812 20:03:57.165226 18379 raft_consensus.cc:493] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.165443 18379 raft_consensus.cc:515] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.165294 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.193s user 0.007s sys 0.010s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5431,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":4847,"dirs.run_cpu_time_us":428,"dirs.run_wall_time_us":34921,"lbm_read_time_us":651,"lbm_reads_lt_1ms":27,"lbm_write_time_us":455,"lbm_writes_lt_1ms":9,"mutex_wait_us":25409,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":51318,"threads_started":2,"update_count":3}
I20260812 20:03:57.165915 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.166258 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.166338 18379 leader_election.cc:290] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.166590 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.166771 18375 tablet_bootstrap.cc:654] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.166929 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.170630 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.170923 18126 raft_consensus.cc:2468] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.171479 17884 leader_election.cc:304] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.171854 18379 raft_consensus.cc:2804] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.171993 18379 raft_consensus.cc:493] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.172102 18379 raft_consensus.cc:3060] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.173825 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.174141 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.174350 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.175115 18379 raft_consensus.cc:515] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.175873 18379 leader_election.cc:290] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.176749 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.176983 18126 raft_consensus.cc:3060] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.177711 18375 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:57.178028 18375 ts_tablet_manager.cc:1403] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260812 20:03:57.179229 18375 raft_consensus.cc:359] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.179594 18375 raft_consensus.cc:385] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.179747 18375 raft_consensus.cc:740] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.180256 18126 raft_consensus.cc:2468] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.180258 18375 consensus_queue.cc:260] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.181342 18418 leader_election.cc:290] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.181928 18375 ts_tablet_manager.cc:1434] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 20:03:57.182159 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.182304 18373 tablet_bootstrap.cc:492] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.182448 18375 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:57.182482 18373 ts_tablet_manager.cc:1403] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.228s user 0.051s sys 0.091s
I20260812 20:03:57.183324 18373 raft_consensus.cc:359] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.183540 18373 raft_consensus.cc:385] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.183652 18373 raft_consensus.cc:740] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.183714 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":5350,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":800,"lbm_read_time_us":615,"lbm_reads_lt_1ms":25,"lbm_write_time_us":392,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":7}
I20260812 20:03:57.183936 18373 consensus_queue.cc:260] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.184499 18373 ts_tablet_manager.cc:1434] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:57.184960 18373 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.185292 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.186000 18375 tablet_bootstrap.cc:654] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.186393 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3950e2b65124f5aa883e4e872b8777f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.187077 18373 tablet_bootstrap.cc:654] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.188495 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f61d5f81507649418fcd65eee57dc3f3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.188675 18281 raft_consensus.cc:3060] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.189961 18375 tablet_bootstrap.cc:492] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:57.190122 18375 ts_tablet_manager.cc:1403] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 20:03:57.191419 18281 raft_consensus.cc:2468] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.191576 18375 raft_consensus.cc:359] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.191783 18375 raft_consensus.cc:385] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.191901 18375 raft_consensus.cc:740] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.192202 18375 consensus_queue.cc:260] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:57.193186 17883 leader_election.cc:343] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:57.193634 17883 leader_election.cc:343] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.193204 18375 ts_tablet_manager.cc:1434] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:57.193806 17883 leader_election.cc:304] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
W20260812 20:03:57.194201 17883 leader_election.cc:343] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:57.194546 17883 leader_election.cc:343] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.194859 18418 raft_consensus.cc:2749] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 20:03:57.198295 17883 consensus_peers.cc:597] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.204110 17884 leader_election.cc:304] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:03:57.204959 18418 raft_consensus.cc:2804] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.205161 18418 raft_consensus.cc:697] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.205485 18418 consensus_queue.cc:237] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.206362 18421 raft_consensus.cc:493] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.206537 18421 raft_consensus.cc:515] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.207494 18421 leader_election.cc:290] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.211673 18373 tablet_bootstrap.cc:492] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.211963 18373 ts_tablet_manager.cc:1403] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.009s
I20260812 20:03:57.212174 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.251s user 0.010s sys 0.010s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5431,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":72441,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":7632,"lbm_read_time_us":22428,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":26,"lbm_write_time_us":494,"lbm_writes_lt_1ms":9,"mutex_wait_us":3973,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":102257,"threads_started":2,"update_count":3}
I20260812 20:03:57.212929 18373 raft_consensus.cc:359] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.213363 18373 raft_consensus.cc:385] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.213466 18373 raft_consensus.cc:740] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.213814 18373 consensus_queue.cc:260] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.214391 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.214494 18373 ts_tablet_manager.cc:1434] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:57.214843 18373 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.215056 18126 raft_consensus.cc:2468] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.217054 18373 tablet_bootstrap.cc:654] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.217646 17884 leader_election.cc:304] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.219364 18418 raft_consensus.cc:2804] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.219551 18418 raft_consensus.cc:493] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.219731 18418 raft_consensus.cc:3060] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.220438 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.231s user 0.005s sys 0.001s Metrics: {"bytes_written":609,"delete_count":0,"lbm_write_time_us":191,"lbm_writes_lt_1ms":4,"mutex_wait_us":222254,"reinsert_count":0,"update_count":7}
I20260812 20:03:57.220695 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.231s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":230439}
I20260812 20:03:57.221043 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.192s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":192334}
I20260812 20:03:57.221168 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.223037 18418 raft_consensus.cc:515] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.223886 18418 leader_election.cc:290] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.225711 18278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.226018 18278 raft_consensus.cc:3060] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.226432 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.105839
I20260812 20:03:57.226943 17768 catalog_manager.cc:5719] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.227156 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.228852 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a00feea243f440d9348756186c6df32" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.229074 18126 raft_consensus.cc:3060] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.232092 18126 raft_consensus.cc:2468] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.239348 17884 leader_election.cc:304] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.240933 18278 raft_consensus.cc:2468] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.242332 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.242342 18377 consensus_queue.cc:1048] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.242681 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.246147 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.246538 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.246922 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.227910 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.247221 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.247423 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.249699 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.250079 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.250346 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.252866 18440 raft_consensus.cc:493] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.253105 18440 raft_consensus.cc:515] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.253293 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":206,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.254240 18440 leader_election.cc:290] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.254616 18381 raft_consensus.cc:2804] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.255076 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.255368 18126 raft_consensus.cc:2468] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:57.255929 18208 leader_election.cc:304] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:57.256451 18440 raft_consensus.cc:2804] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.256610 18440 raft_consensus.cc:493] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.256722 18440 raft_consensus.cc:3060] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.257404 17768 catalog_manager.cc:5719] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.261340 18440 raft_consensus.cc:515] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.261770 18381 raft_consensus.cc:697] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.262163 18381 consensus_queue.cc:237] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.262349 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.262655 17962 raft_consensus.cc:2468] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:57.267253 18421 raft_consensus.cc:493] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.267491 18421 raft_consensus.cc:515] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.268167 18421 leader_election.cc:290] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.268626 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.268851 18127 raft_consensus.cc:2468] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.269400 17884 leader_election.cc:304] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.271322 18421 raft_consensus.cc:2804] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.271440 18421 raft_consensus.cc:493] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.271553 18421 raft_consensus.cc:3060] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.271694 18379 consensus_queue.cc:1048] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.275499 18421 raft_consensus.cc:515] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.276201 18421 leader_election.cc:290] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.276733 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.276978 18127 raft_consensus.cc:3060] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.279379 18127 raft_consensus.cc:2468] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.279945 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.279961 17884 leader_election.cc:304] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.280365 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.280375 18379 raft_consensus.cc:2804] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.280537 18379 raft_consensus.cc:697] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:57.280848 18379 consensus_queue.cc:237] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.280956 18377 raft_consensus.cc:493] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.281129 18377 raft_consensus.cc:515] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.281796 18377 leader_election.cc:290] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.282510 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.282742 17962 raft_consensus.cc:2468] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:57.283278 18043 leader_election.cc:304] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.283620 18377 raft_consensus.cc:2804] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:57.283710 18377 raft_consensus.cc:493] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:57.283814 18377 raft_consensus.cc:3060] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.286859 18377 raft_consensus.cc:515] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.287628 18377 leader_election.cc:290] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.288632 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:57.288928 17962 raft_consensus.cc:3060] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.289108 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":194,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.290383 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.290843 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.145019
I20260812 20:03:57.291170 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.145019
I20260812 20:03:57.292259 17962 raft_consensus.cc:2468] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:57.293033 18043 leader_election.cc:304] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:57.293517 18377 raft_consensus.cc:2804] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.294476 18377 raft_consensus.cc:697] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:57.294849 18377 consensus_queue.cc:237] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.297999 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:57.298414 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.298707 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.299654 17768 catalog_manager.cc:5719] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.270087 18278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.312662 18278 raft_consensus.cc:2468] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:57.276818 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abdf5914e4334bfe9ea2ef52b14f0716" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:57.313090 18279 raft_consensus.cc:3060] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.314116 18440 leader_election.cc:290] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.314543 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:57.314805 18126 raft_consensus.cc:3060] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.316717 18279 raft_consensus.cc:2468] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:57.278148 18373 tablet_bootstrap.cc:492] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.317094 18373 ts_tablet_manager.cc:1403] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.102s user 0.019s sys 0.032s
I20260812 20:03:57.317890 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba44ce3aac624af697574a4274f38bf5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:57.318116 17962 raft_consensus.cc:3060] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.318334 18126 raft_consensus.cc:2468] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:57.318938 18208 leader_election.cc:304] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:57.319308 18440 raft_consensus.cc:2804] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:57.319836 18440 raft_consensus.cc:697] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:57.320178 18440 consensus_queue.cc:237] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.321143 17962 raft_consensus.cc:2468] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:57.322736 18373 raft_consensus.cc:359] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.322983 18373 raft_consensus.cc:385] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.323083 18373 raft_consensus.cc:740] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.323174 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.323352 18373 consensus_queue.cc:260] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.323657 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.323936 18373 ts_tablet_manager.cc:1434] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260812 20:03:57.323938 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.324266 18373 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.326519 18373 tablet_bootstrap.cc:654] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.288717 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:57.288745 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
W20260812 20:03:57.329460 18043 leader_election.cc:343] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:57.329900 18043 leader_election.cc:343] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.330358 18377 consensus_queue.cc:1048] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.331176 18418 consensus_queue.cc:1048] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.338681 17767 catalog_manager.cc:5719] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:57.349196 18406 consensus_queue.cc:1048] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.353585 18373 tablet_bootstrap.cc:492] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.353864 18373 ts_tablet_manager.cc:1403] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.004s
I20260812 20:03:57.354841 18373 raft_consensus.cc:359] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.355094 18373 raft_consensus.cc:385] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.355180 18373 raft_consensus.cc:740] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.355463 18373 consensus_queue.cc:260] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.356035 18373 ts_tablet_manager.cc:1434] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:57.356473 18373 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.361920 18418 consensus_queue.cc:1048] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.363528 18373 tablet_bootstrap.cc:654] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:57.368108 17883 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.14.3:34495, user_credentials={real_user=slave}} blocked reactor thread for 68599.5us
I20260812 20:03:57.369102 18481 consensus_queue.cc:1048] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.377422 18373 tablet_bootstrap.cc:492] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.377650 18373 ts_tablet_manager.cc:1403] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20260812 20:03:57.378682 18373 raft_consensus.cc:359] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.378931 18373 raft_consensus.cc:385] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.379037 18373 raft_consensus.cc:740] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.380432 17767 catalog_manager.cc:5719] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.381809 18373 consensus_queue.cc:260] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.382707 18373 ts_tablet_manager.cc:1434] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260812 20:03:57.383117 18373 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.385274 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":60857}
I20260812 20:03:57.385485 18373 tablet_bootstrap.cc:654] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.385672 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":61227}
I20260812 20:03:57.386300 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.062s user 0.007s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":192,"lbm_writes_lt_1ms":4,"mutex_wait_us":53859,"reinsert_count":0,"update_count":2}
I20260812 20:03:57.386978 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.096s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5771,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":11561,"lbm_read_time_us":690,"lbm_reads_lt_1ms":27,"lbm_write_time_us":374,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260812 20:03:57.388063 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.167s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":5350,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":685,"lbm_reads_lt_1ms":25,"lbm_write_time_us":477,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":7}
I20260812 20:03:57.389824 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
W20260812 20:03:57.396946 17883 consensus_peers.cc:597] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.402096 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.402599 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.402961 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.405257 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.405609 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.405967 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201906
I20260812 20:03:57.406363 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201906
I20260812 20:03:57.406842 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:57.406746 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.407305 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.412734 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.170s user 0.003s sys 0.001s Metrics: {"bytes_written":513,"delete_count":0,"lbm_write_time_us":169,"lbm_writes_lt_1ms":4,"mutex_wait_us":147160,"reinsert_count":0,"update_count":4}
I20260812 20:03:57.413424 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212207
I20260812 20:03:57.409600 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":196,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.418408 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.170s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":170311}
I20260812 20:03:57.419541 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":27794}
I20260812 20:03:57.420342 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.177s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":181,"lbm_writes_lt_1ms":4,"mutex_wait_us":169499,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.425923 17768 catalog_manager.cc:5719] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:57.427700 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5875,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":57,"lbm_read_time_us":527,"lbm_reads_lt_1ms":27,"lbm_write_time_us":2978,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":8,"mutex_wait_us":2875,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260812 20:03:57.431567 18373 tablet_bootstrap.cc:492] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.431804 18373 ts_tablet_manager.cc:1403] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20260812 20:03:57.432749 18373 raft_consensus.cc:359] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.433081 18373 raft_consensus.cc:385] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.433182 18373 raft_consensus.cc:740] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.433524 18373 consensus_queue.cc:260] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.436825 18373 ts_tablet_manager.cc:1434] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 20:03:57.437844 18373 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.439913 18373 tablet_bootstrap.cc:654] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.440554 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.440465 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.007s sys 0.005s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5848,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":100,"lbm_read_time_us":561,"lbm_reads_lt_1ms":31,"lbm_write_time_us":5297,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":8,"mutex_wait_us":3760,"peak_mem_usage":32768,"reinsert_count":0,"update_count":5}
I20260812 20:03:57.441074 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.441092 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.441406 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.441490 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.441725 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.441917 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.442037 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.443848 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.444186 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.444480 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.444480 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.444855 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.444922 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.448133 18373 tablet_bootstrap.cc:492] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.448292 18373 ts_tablet_manager.cc:1403] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260812 20:03:57.448282 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":234,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.449297 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":132,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:57.449317 18373 raft_consensus.cc:359] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.449579 18373 raft_consensus.cc:385] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.449750 18373 raft_consensus.cc:740] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.450017 18373 consensus_queue.cc:260] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.450611 18373 ts_tablet_manager.cc:1434] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:57.451607 18379 consensus_queue.cc:1048] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.456492 18379 consensus_queue.cc:1048] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.460574 18379 consensus_queue.cc:1048] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.465888 18279 raft_consensus.cc:3060] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.470290 18379 consensus_queue.cc:1048] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.471122 18126 raft_consensus.cc:3060] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.475188 18379 consensus_queue.cc:1048] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.481678 18381 consensus_queue.cc:1048] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.482374 18440 consensus_queue.cc:1048] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.486104 18386 consensus_queue.cc:1048] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.503461 18443 raft_consensus.cc:493] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.503686 18443 raft_consensus.cc:515] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.504630 18443 leader_election.cc:290] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.505259 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.509548 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fbf74c9ba7470ea946e2b437b3d71e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.509889 17964 raft_consensus.cc:2393] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate d222b63b24184ca6bc530a1547fe3297 in this term.
I20260812 20:03:57.510504 18207 leader_election.cc:304] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.510936 18443 raft_consensus.cc:3060] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.513675 18443 raft_consensus.cc:2749] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.526759 18443 raft_consensus.cc:493] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.527020 18443 raft_consensus.cc:515] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.527757 18443 leader_election.cc:290] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.528491 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.528529 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecfbe43f35904b94b26f8e6a239cf393" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.528870 18126 raft_consensus.cc:2393] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:03:57.529420 18208 leader_election.cc:304] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.529845 18443 raft_consensus.cc:3060] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.531905 18443 raft_consensus.cc:2749] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.538034 17767 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-0" } modify_external_catalogs: true
I20260812 20:03:57.538328 17767 catalog_manager.cc:2781] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-0" } modify_external_catalogs: true
I20260812 20:03:57.538772 18381 consensus_queue.cc:1048] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.555823 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.556173 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.556988 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.557421 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.557601 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.557777 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.558092 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.559639 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.559957 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.559921 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.560264 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.560314 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.560622 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.560856 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.561184 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 11d7acaf0d9645ada0116c85aaac2c40
I20260812 20:03:57.563773 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.003s sys 0.001s Metrics: {"bytes_written":923,"delete_count":0,"lbm_write_time_us":230,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:03:57.567863 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.569176 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.569736 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.219264
I20260812 20:03:57.570241 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.570322 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.570557 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.570693 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:57.570955 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.574116 18501 tablet_replica.cc:333] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:57.574622 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.574891 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet abdf5914e4334bfe9ea2ef52b14f0716
I20260812 20:03:57.575414 18501 raft_consensus.cc:2243] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.575882 18501 raft_consensus.cc:2272] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.576822 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.576915 18502 tablet_replica.cc:333] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.577534 18502 raft_consensus.cc:2243] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.578022 18502 raft_consensus.cc:2272] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.578120 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.578361 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5637eaa46a444685a84db3a4a50c0e3b
I20260812 20:03:57.579082 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.006s sys 0.000s Metrics: {"bytes_written":923,"delete_count":0,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:03:57.579793 18501 ts_tablet_manager.cc:1916] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.579802 18443 raft_consensus.cc:493] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.579955 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.579965 18443 raft_consensus.cc:515] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.580714 18502 ts_tablet_manager.cc:1916] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.581038 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 95f1ea51abb743b7b7b6b5feac8af23b
I20260812 20:03:57.581216 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:57.581543 17964 raft_consensus.cc:2393] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate d222b63b24184ca6bc530a1547fe3297 in this term.
I20260812 20:03:57.581527 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6443,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":3703,"lbm_read_time_us":2025,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":26,"lbm_write_time_us":330,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
I20260812 20:03:57.580874 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.582360 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.583187 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 2f51859318ec4174be3dd534658149c2
I20260812 20:03:57.583217 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.583617 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8f649dddfa5466988e196d03656eb6f" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:57.584316 18208 leader_election.cc:304] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:57.584692 18499 raft_consensus.cc:3060] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:57.584830 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f51859318ec4174be3dd534658149c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.585932 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f51859318ec4174be3dd534658149c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.585991 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.586196 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet e3950e2b65124f5aa883e4e872b8777f
I20260812 20:03:57.586232 17932 tablet_service.cc:1558] Processing DeleteTablet for tablet 5637eaa46a444685a84db3a4a50c0e3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.586055 17922 tablet_service.cc:1558] Processing DeleteTablet for tablet abdf5914e4334bfe9ea2ef52b14f0716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.586697 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f51859318ec4174be3dd534658149c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.586966 17932 tablet_service.cc:1558] Processing DeleteTablet for tablet e3950e2b65124f5aa883e4e872b8777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.587065 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 95f1ea51abb743b7b7b6b5feac8af23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.587255 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 11d7acaf0d9645ada0116c85aaac2c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.587598 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet e3950e2b65124f5aa883e4e872b8777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.588419 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet e3950e2b65124f5aa883e4e872b8777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.588640 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5a00feea243f440d9348756186c6df32
I20260812 20:03:57.589323 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 5a00feea243f440d9348756186c6df32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.589856 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 5a00feea243f440d9348756186c6df32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.590204 18503 tablet_replica.cc:333] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:57.590768 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 5a00feea243f440d9348756186c6df32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.590798 18502 ts_tablet_manager.cc:1929] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.590994 18502 log.cc:1199] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/11d7acaf0d9645ada0116c85aaac2c40
I20260812 20:03:57.591030 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b8f649dddfa5466988e196d03656eb6f
I20260812 20:03:57.591454 18503 raft_consensus.cc:2243] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:57.591878 18502 ts_tablet_manager.cc:1950] T 11d7acaf0d9645ada0116c85aaac2c40 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.592604 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet b8f649dddfa5466988e196d03656eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.592711 18501 ts_tablet_manager.cc:1929] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.592873 18501 log.cc:1199] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:57.591787 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet b8f649dddfa5466988e196d03656eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.593394 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet b8f649dddfa5466988e196d03656eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.593677 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:57.593858 18503 raft_consensus.cc:2272] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.594539 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.594568 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 11d7acaf0d9645ada0116c85aaac2c40 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.594993 18443 leader_election.cc:290] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:57.595047 18502 tablet_replica.cc:333] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.595124 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.595304 18502 raft_consensus.cc:2243] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.595569 18502 raft_consensus.cc:2272] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.596035 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet ecfbe43f35904b94b26f8e6a239cf393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.596275 18499 raft_consensus.cc:2749] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.596304 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet d50211ef1e684ab4824281af94b9febe
I20260812 20:03:57.596607 18502 ts_tablet_manager.cc:1916] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.596526 18406 consensus_queue.cc:1048] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.597455 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet d50211ef1e684ab4824281af94b9febe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.598217 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet d50211ef1e684ab4824281af94b9febe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.598425 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet d50211ef1e684ab4824281af94b9febe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.598449 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 530fa3bb1bdf4fa083a35406188e2582
I20260812 20:03:57.599104 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.599412 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.599582 18381 consensus_queue.cc:1048] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.600178 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 530fa3bb1bdf4fa083a35406188e2582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.600463 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet f7c30fa50d4049c2a3370881418edf80
I20260812 20:03:57.601501 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet f7c30fa50d4049c2a3370881418edf80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.601754 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet f7c30fa50d4049c2a3370881418edf80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.602451 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet f7c30fa50d4049c2a3370881418edf80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.602705 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 07fbf74c9ba7470ea946e2b437b3d71e
I20260812 20:03:57.603727 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.604578 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.604671 18502 ts_tablet_manager.cc:1929] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.604835 18502 log.cc:1199] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/abdf5914e4334bfe9ea2ef52b14f0716
I20260812 20:03:57.604959 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet f61d5f81507649418fcd65eee57dc3f3
I20260812 20:03:57.605379 18502 ts_tablet_manager.cc:1950] T abdf5914e4334bfe9ea2ef52b14f0716 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.606017 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet f61d5f81507649418fcd65eee57dc3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.606155 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet f61d5f81507649418fcd65eee57dc3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.606650 18502 tablet_replica.cc:333] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.606774 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet abdf5914e4334bfe9ea2ef52b14f0716 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.606835 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 07fbf74c9ba7470ea946e2b437b3d71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.607028 18502 raft_consensus.cc:2243] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:57.607568 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet f61d5f81507649418fcd65eee57dc3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.607721 18502 raft_consensus.cc:2272] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.607882 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet ba44ce3aac624af697574a4274f38bf5
I20260812 20:03:57.608261 18501 ts_tablet_manager.cc:1950] T ecfbe43f35904b94b26f8e6a239cf393 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:57.608703 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet ba44ce3aac624af697574a4274f38bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.608901 18502 ts_tablet_manager.cc:1916] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.608964 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet ba44ce3aac624af697574a4274f38bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.609853 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet ba44ce3aac624af697574a4274f38bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.610113 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet c71fd265588a4fccae425a41ef98fb05
I20260812 20:03:57.611616 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet c71fd265588a4fccae425a41ef98fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:57.612478 18503 ts_tablet_manager.cc:1916] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.613118 18255 tablet_service.cc:1558] Processing DeleteTablet for tablet c71fd265588a4fccae425a41ef98fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:57.616219 18502 ts_tablet_manager.cc:1929] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.616444 18502 log.cc:1199] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5637eaa46a444685a84db3a4a50c0e3b
I20260812 20:03:57.616761 18421 consensus_queue.cc:1048] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.617164 18502 ts_tablet_manager.cc:1950] T 5637eaa46a444685a84db3a4a50c0e3b P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.618340 18502 tablet_replica.cc:333] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.618646 18502 raft_consensus.cc:2243] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.618757 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5637eaa46a444685a84db3a4a50c0e3b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.618749 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet c71fd265588a4fccae425a41ef98fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:57 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:57.618991 18502 raft_consensus.cc:2272] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.619985 18501 tablet_replica.cc:333] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:57.620245 18501 raft_consensus.cc:2243] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.620466 18501 raft_consensus.cc:2272] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.620680 17769 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38702:
name: "table-1"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260812 20:03:57.621470 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.051s user 0.004s sys 0.008s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6443,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":568,"lbm_reads_lt_1ms":27,"lbm_write_time_us":350,"lbm_writes_lt_1ms":9,"mutex_wait_us":1290,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
W20260812 20:03:57.621783 17769 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 20:03:57.622328 18502 ts_tablet_manager.cc:1916] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.625605 18406 consensus_queue.cc:1048] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.628253 18501 ts_tablet_manager.cc:1916] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.630072 18502 ts_tablet_manager.cc:1929] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.630232 18502 log.cc:1199] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/95f1ea51abb743b7b7b6b5feac8af23b
I20260812 20:03:57.630632 18502 ts_tablet_manager.cc:1950] T 95f1ea51abb743b7b7b6b5feac8af23b P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.631796 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 95f1ea51abb743b7b7b6b5feac8af23b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.632220 18502 tablet_replica.cc:333] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.632428 18502 raft_consensus.cc:2243] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.632673 18502 raft_consensus.cc:2272] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.633860 18502 ts_tablet_manager.cc:1916] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.635322 18503 ts_tablet_manager.cc:1929] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.638724 18501 ts_tablet_manager.cc:1929] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.639039 18503 log.cc:1199] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:57.639820 18503 ts_tablet_manager.cc:1950] T ecfbe43f35904b94b26f8e6a239cf393 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:57.643518 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.644966 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.649936 18502 ts_tablet_manager.cc:1929] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.650142 18502 log.cc:1199] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/2f51859318ec4174be3dd534658149c2
I20260812 20:03:57.650643 18502 ts_tablet_manager.cc:1950] T 2f51859318ec4174be3dd534658149c2 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.651978 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 2f51859318ec4174be3dd534658149c2 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.652341 18502 tablet_replica.cc:333] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.652583 18502 raft_consensus.cc:2243] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.652773 18502 raft_consensus.cc:2272] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.658017 18502 ts_tablet_manager.cc:1916] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.659614 18501 log.cc:1199] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/11d7acaf0d9645ada0116c85aaac2c40
I20260812 20:03:57.660246 18501 ts_tablet_manager.cc:1950] T 11d7acaf0d9645ada0116c85aaac2c40 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:57.662499 18503 tablet_replica.cc:333] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:57.662840 18503 raft_consensus.cc:2243] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:57.669044 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.008s sys 0.004s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1135,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1200,"lbm_writes_lt_1ms":27,"mutex_wait_us":3,"peak_mem_usage":0,"rows_written":17}
I20260812 20:03:57.670187 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:57.670581 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:57.670907 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.671027 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 11d7acaf0d9645ada0116c85aaac2c40 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.671257 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.671937 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20260812 20:03:57.672024 18255 tablet_service.cc:1511] Processing CreateTablet for tablet a2950ec0320c432887522c75de50d8c5 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.672346 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.672501 18255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2950ec0320c432887522c75de50d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.673882 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.674865 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.924971
I20260812 20:03:57.675670 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.924971
I20260812 20:03:57.679376 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.679955 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5c8e0a860641168abfaee4c7216fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.679878 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.680163 17921 tablet_service.cc:1511] Processing CreateTablet for tablet a2950ec0320c432887522c75de50d8c5 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.680393 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5c8e0a860641168abfaee4c7216fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.680789 17932 tablet_service.cc:1511] Processing CreateTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.681155 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad9b9e5ec944a1aa309cc9332b2dbca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.683239 18502 ts_tablet_manager.cc:1929] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.683414 18502 log.cc:1199] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/e3950e2b65124f5aa883e4e872b8777f
I20260812 20:03:57.683414 18089 tablet_service.cc:1511] Processing CreateTablet for tablet bb77a5184b8a459db4499e77cd2df733 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.683825 18502 ts_tablet_manager.cc:1950] T e3950e2b65124f5aa883e4e872b8777f P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.683867 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb77a5184b8a459db4499e77cd2df733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.685101 18502 tablet_replica.cc:333] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.685300 18502 raft_consensus.cc:2243] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.685496 18502 raft_consensus.cc:2272] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.685990 17931 tablet_service.cc:1511] Processing CreateTablet for tablet bb77a5184b8a459db4499e77cd2df733 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.686231 18502 ts_tablet_manager.cc:1916] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.686406 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb77a5184b8a459db4499e77cd2df733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.686815 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2950ec0320c432887522c75de50d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.687089 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 489ea23ba78a47bda493198355f9b340 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.687489 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489ea23ba78a47bda493198355f9b340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.690164 18248 tablet_service.cc:1511] Processing CreateTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.690631 18248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5c8e0a860641168abfaee4c7216fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.694535 18240 tablet_service.cc:1511] Processing CreateTablet for tablet bb77a5184b8a459db4499e77cd2df733 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.694692 18508 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:57.695030 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb77a5184b8a459db4499e77cd2df733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.695237 18503 pending_rounds.cc:70] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Trying to abort 1 pending ops.
I20260812 20:03:57.695472 18503 pending_rounds.cc:77] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770392705724416 op_type: NO_OP noop_request { }
I20260812 20:03:57.695658 18503 raft_consensus.cc:2889] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:03:57.695804 18503 raft_consensus.cc:2272] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.696255 18501 tablet_replica.cc:333] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:57.696287 18502 ts_tablet_manager.cc:1929] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.696434 18502 log.cc:1199] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5a00feea243f440d9348756186c6df32
I20260812 20:03:57.696521 18501 raft_consensus.cc:2243] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.696729 18508 tablet_bootstrap.cc:654] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.696753 18501 raft_consensus.cc:2272] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.696899 18502 ts_tablet_manager.cc:1950] T 5a00feea243f440d9348756186c6df32 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.698063 18251 tablet_service.cc:1511] Processing CreateTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.679369 18105 tablet_service.cc:1511] Processing CreateTablet for tablet a2950ec0320c432887522c75de50d8c5 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.699240 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2950ec0320c432887522c75de50d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.706755 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.011s sys 0.004s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":224,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1135,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1117,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260812 20:03:57.707908 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:57.708249 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:57.708635 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.708976 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:57.709659 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20260812 20:03:57.710153 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.712304 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.713332 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.926568
I20260812 20:03:57.714450 18406 consensus_queue.cc:1048] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 20:03:57.715492 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.043s user 0.006s sys 0.000s Metrics: {"bytes_written":713,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260812 20:03:57.715904 18502 tablet_replica.cc:333] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.716130 18502 raft_consensus.cc:2243] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.716328 18502 raft_consensus.cc:2272] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.717574 18502 ts_tablet_manager.cc:1916] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.717859 18373 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.718194 18251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad9b9e5ec944a1aa309cc9332b2dbca. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:03:57.723052 17883 consensus_peers.cc:597] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a00feea243f440d9348756186c6df32. This is attempt 1: this message will repeat every 5th retry.
W20260812 20:03:57.723708 18043 consensus_peers.cc:597] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.726012 18094 tablet_service.cc:1511] Processing CreateTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.726475 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad9b9e5ec944a1aa309cc9332b2dbca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.728593 18421 consensus_queue.cc:1048] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.728749 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet e3950e2b65124f5aa883e4e872b8777f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.733705 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5a00feea243f440d9348756186c6df32 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.733667 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.000s sys 0.003s Metrics: {"bytes_written":713,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260812 20:03:57.745324 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 489ea23ba78a47bda493198355f9b340 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.746249 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489ea23ba78a47bda493198355f9b340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.746443 18499 consensus_queue.cc:1048] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260812 20:03:57.753233 18373 tablet_bootstrap.cc:654] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:57.765130 17883 consensus_peers.cc:597] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3950e2b65124f5aa883e4e872b8777f. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.771005 18501 ts_tablet_manager.cc:1916] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.772359 18502 ts_tablet_manager.cc:1929] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.772609 18502 log.cc:1199] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b8f649dddfa5466988e196d03656eb6f
I20260812 20:03:57.775440 18502 ts_tablet_manager.cc:1950] T b8f649dddfa5466988e196d03656eb6f P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.776033 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1858 bytes on disk
I20260812 20:03:57.776444 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1858 bytes on disk
I20260812 20:03:57.776760 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1858 bytes on disk
I20260812 20:03:57.777100 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b8f649dddfa5466988e196d03656eb6f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.777551 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20260812 20:03:57.778061 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.778436 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:57.779138 18240 tablet_service.cc:1511] Processing CreateTablet for tablet 489ea23ba78a47bda493198355f9b340 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.779681 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489ea23ba78a47bda493198355f9b340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.779856 18503 ts_tablet_manager.cc:1916] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.780573 17937 tablet_service.cc:1511] Processing CreateTablet for tablet c5c9d056274f4c1ab24c5f317981d016 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.781116 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c9d056274f4c1ab24c5f317981d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.785642 18508 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.785825 18508 ts_tablet_manager.cc:1403] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.091s user 0.034s sys 0.047s
I20260812 20:03:57.786912 18508 raft_consensus.cc:359] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.787159 18508 raft_consensus.cc:385] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.787256 18508 raft_consensus.cc:740] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.787515 18508 consensus_queue.cc:260] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.787688 18502 tablet_replica.cc:333] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.787994 18502 raft_consensus.cc:2243] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.788250 18502 raft_consensus.cc:2272] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.788797 18508 ts_tablet_manager.cc:1434] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 20:03:57.789105 18508 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:57.789872 18502 ts_tablet_manager.cc:1916] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.793447 18098 tablet_service.cc:1511] Processing CreateTablet for tablet c5c9d056274f4c1ab24c5f317981d016 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.793722 18508 tablet_bootstrap.cc:654] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.796795 18503 ts_tablet_manager.cc:1929] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.797086 18503 log.cc:1199] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/abdf5914e4334bfe9ea2ef52b14f0716
I20260812 20:03:57.797739 18503 ts_tablet_manager.cc:1950] T abdf5914e4334bfe9ea2ef52b14f0716 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:57.798219 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c9d056274f4c1ab24c5f317981d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.799919 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet abdf5914e4334bfe9ea2ef52b14f0716 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.800549 18503 tablet_replica.cc:333] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:57.800758 18502 ts_tablet_manager.cc:1929] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.800963 18502 log.cc:1199] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:57.800987 18503 raft_consensus.cc:2243] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.801316 18503 raft_consensus.cc:2272] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.801434 18502 ts_tablet_manager.cc:1950] T ecfbe43f35904b94b26f8e6a239cf393 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.802971 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet ecfbe43f35904b94b26f8e6a239cf393 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.805073 18502 tablet_replica.cc:333] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.805029 18094 tablet_service.cc:1511] Processing CreateTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.804982 18240 tablet_service.cc:1511] Processing CreateTablet for tablet c5c9d056274f4c1ab24c5f317981d016 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.805334 18502 raft_consensus.cc:2243] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.805585 18502 raft_consensus.cc:2272] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.805850 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c9d056274f4c1ab24c5f317981d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.807685 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b607a7c7ca104dbca81ab5ddae140a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.808368 18502 ts_tablet_manager.cc:1916] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.808525 18503 ts_tablet_manager.cc:1916] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.810097 17931 tablet_service.cc:1511] Processing CreateTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.810671 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b607a7c7ca104dbca81ab5ddae140a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.817905 18406 consensus_queue.cc:1048] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.824328 18515 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:57.824434 18502 ts_tablet_manager.cc:1929] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.824586 18502 log.cc:1199] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/d50211ef1e684ab4824281af94b9febe
I20260812 20:03:57.825166 18502 ts_tablet_manager.cc:1950] T d50211ef1e684ab4824281af94b9febe P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.830835 18502 tablet_replica.cc:333] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:57.831174 18502 raft_consensus.cc:2243] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.831425 18502 raft_consensus.cc:2272] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:57.831647 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet d50211ef1e684ab4824281af94b9febe (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.833686 18502 ts_tablet_manager.cc:1916] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:57.837998 18251 tablet_service.cc:1511] Processing CreateTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.838532 18251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b607a7c7ca104dbca81ab5ddae140a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.838065 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.839159 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb8accc87ef4ee4a5186719c042db42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.842741 18499 consensus_queue.cc:1048] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.844116 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.844689 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb8accc87ef4ee4a5186719c042db42. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:03:57.848817 18043 consensus_peers.cc:597] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f51859318ec4174be3dd534658149c2. This is attempt 1: this message will repeat every 5th retry.
W20260812 20:03:57.862826 17884 consensus_peers.cc:597] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11d7acaf0d9645ada0116c85aaac2c40. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.866588 18373 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:57.866881 18373 ts_tablet_manager.cc:1403] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.009s
I20260812 20:03:57.867947 18373 raft_consensus.cc:359] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:57.868160 18373 raft_consensus.cc:385] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.868228 18373 raft_consensus.cc:740] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.868477 18373 consensus_queue.cc:260] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:57.869031 18373 ts_tablet_manager.cc:1434] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:57.869336 18373 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:57.874329 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.874850 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1ec98f1d914666bc5e02306ee16d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.875357 18248 tablet_service.cc:1511] Processing CreateTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.875399 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.875780 18248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1ec98f1d914666bc5e02306ee16d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.876799 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 5000368644a64059af5235482fe2e79c (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.878049 18418 consensus_queue.cc:1048] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:57.878796 18515 tablet_bootstrap.cc:654] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.879484 17932 tablet_service.cc:1511] Processing CreateTablet for tablet 5000368644a64059af5235482fe2e79c (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.879931 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5000368644a64059af5235482fe2e79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.880719 18251 tablet_service.cc:1511] Processing CreateTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.881186 18251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb8accc87ef4ee4a5186719c042db42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.885661 18373 tablet_bootstrap.cc:654] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:57.887375 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5000368644a64059af5235482fe2e79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.888006 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1ec98f1d914666bc5e02306ee16d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.890588 18255 tablet_service.cc:1511] Processing CreateTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.891108 18255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7e3cf6c3324a42aa08da0128dda7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.893929 18104 tablet_service.cc:1511] Processing CreateTablet for tablet 9f052b8cb7864b5693be4845f8c473ac (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.894435 18104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f052b8cb7864b5693be4845f8c473ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.895653 18244 tablet_service.cc:1511] Processing CreateTablet for tablet 9f052b8cb7864b5693be4845f8c473ac (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.896166 18244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f052b8cb7864b5693be4845f8c473ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.899394 18100 tablet_service.cc:1511] Processing CreateTablet for tablet 6397453a07df4814be9827bb15738081 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.899873 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6397453a07df4814be9827bb15738081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.900347 18249 tablet_service.cc:1511] Processing CreateTablet for tablet 6397453a07df4814be9827bb15738081 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.900813 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6397453a07df4814be9827bb15738081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.905496 18243 tablet_service.cc:1511] Processing CreateTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.905941 18243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77032e8e8636495a9dfdfa1ce5d85021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.906816 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.907222 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77032e8e8636495a9dfdfa1ce5d85021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.910036 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.910461 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfafbefa0be43d694d032f733f1d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.911528 18242 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.911909 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfafbefa0be43d694d032f733f1d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.913949 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 44959dd27d9642f0ad8ddf8395887083 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.914408 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44959dd27d9642f0ad8ddf8395887083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.914780 18257 tablet_service.cc:1511] Processing CreateTablet for tablet 44959dd27d9642f0ad8ddf8395887083 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.915194 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44959dd27d9642f0ad8ddf8395887083. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:03:57.920914 18043 consensus_peers.cc:597] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:57.933693 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1858 bytes on disk
I20260812 20:03:57.934383 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20260812 20:03:57.943979 18248 tablet_service.cc:1511] Processing CreateTablet for tablet 5000368644a64059af5235482fe2e79c (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.944624 18248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5000368644a64059af5235482fe2e79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.948288 18418 raft_consensus.cc:493] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:57.948509 18418 raft_consensus.cc:515] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:57.949421 18418 leader_election.cc:290] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:57.950376 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:57.951861 17883 leader_election.cc:343] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.954275 17922 tablet_service.cc:1511] Processing CreateTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.954845 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77032e8e8636495a9dfdfa1ce5d85021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.955860 17921 tablet_service.cc:1511] Processing CreateTablet for tablet 6397453a07df4814be9827bb15738081 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.956310 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6397453a07df4814be9827bb15738081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.956576 17756 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:57.958139 17929 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.958635 17929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfafbefa0be43d694d032f733f1d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.959790 17932 tablet_service.cc:1511] Processing CreateTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.960183 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7e3cf6c3324a42aa08da0128dda7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.961165 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.283s user 0.005s sys 0.012s Metrics: {"dirs.queue_time_us":412,"dirs.run_cpu_time_us":800,"dirs.run_wall_time_us":3387,"lbm_write_time_us":478,"lbm_writes_lt_1ms":7,"mutex_wait_us":131,"num_input_rowsets":0}
I20260812 20:03:57.962066 17924 tablet_service.cc:1511] Processing CreateTablet for tablet 44959dd27d9642f0ad8ddf8395887083 (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.962530 17924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44959dd27d9642f0ad8ddf8395887083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.962441 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.286s user 0.036s sys 0.008s Metrics: {"bytes_written":18500,"cfile_cache_hit":23,"cfile_cache_hit_bytes":9409,"cfile_cache_miss":59,"cfile_cache_miss_bytes":16055,"cfile_init":16,"delta_iterators_relevant":17,"dirs.queue_time_us":23254,"dirs.run_cpu_time_us":1818,"dirs.run_wall_time_us":16281,"drs_written":1,"lbm_read_time_us":7317,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":122,"lbm_write_time_us":1187,"lbm_writes_lt_1ms":27,"mutex_wait_us":251,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":37,"thread_start_us":20834,"threads_started":3}
I20260812 20:03:57.966964 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:57.968734 17884 leader_election.cc:343] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:57.968943 17884 leader_election.cc:304] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:57.969596 18381 raft_consensus.cc:2749] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:57.979148 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 9f052b8cb7864b5693be4845f8c473ac (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.979413 18508 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:57.979612 18508 ts_tablet_manager.cc:1403] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.191s user 0.011s sys 0.047s
I20260812 20:03:57.979699 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f052b8cb7864b5693be4845f8c473ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.980680 18508 raft_consensus.cc:359] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:57.981225 18501 ts_tablet_manager.cc:1929] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:57.981328 18508 raft_consensus.cc:385] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:57.981432 18508 raft_consensus.cc:740] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:57.981436 18501 log.cc:1199] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/abdf5914e4334bfe9ea2ef52b14f0716
I20260812 20:03:57.982107 18501 ts_tablet_manager.cc:1950] T abdf5914e4334bfe9ea2ef52b14f0716 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:57.983705 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet abdf5914e4334bfe9ea2ef52b14f0716 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:57.985396 18508 consensus_queue.cc:260] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:57.985729 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca (DEFAULT_TABLE table=table-1 [id=b428b54d8c7e45a1826c075891c20732]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:57.985991 18508 ts_tablet_manager.cc:1434] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 20:03:57.986300 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7e3cf6c3324a42aa08da0128dda7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:57.994356 18508 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:57.995585 18502 ts_tablet_manager.cc:1929] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:57.995752 18502 log.cc:1199] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/530fa3bb1bdf4fa083a35406188e2582
I20260812 20:03:57.996299 18502 ts_tablet_manager.cc:1950] T 530fa3bb1bdf4fa083a35406188e2582 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:57.996865 18501 tablet_replica.cc:333] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:57.997143 18501 raft_consensus.cc:2243] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:57.997399 18501 raft_consensus.cc:2272] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 20:03:58.002177 17883 consensus_peers.cc:597] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11d7acaf0d9645ada0116c85aaac2c40. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.005916 18501 ts_tablet_manager.cc:1916] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.010982 18515 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.011000 18508 tablet_bootstrap.cc:654] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.011178 18515 ts_tablet_manager.cc:1403] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.187s user 0.001s sys 0.003s
I20260812 20:03:58.012092 18515 raft_consensus.cc:359] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.012308 18515 raft_consensus.cc:385] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.012408 18515 raft_consensus.cc:740] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.012701 18515 consensus_queue.cc:260] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.012952 18501 ts_tablet_manager.cc:1929] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.013113 18501 log.cc:1199] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5637eaa46a444685a84db3a4a50c0e3b
I20260812 20:03:58.016364 18503 ts_tablet_manager.cc:1929] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.016661 18503 log.cc:1199] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5637eaa46a444685a84db3a4a50c0e3b
I20260812 20:03:58.017310 18503 ts_tablet_manager.cc:1950] T 5637eaa46a444685a84db3a4a50c0e3b P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.019083 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5637eaa46a444685a84db3a4a50c0e3b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.019279 18515 ts_tablet_manager.cc:1434] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260812 20:03:58.021932 18503 tablet_replica.cc:333] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.024672 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 530fa3bb1bdf4fa083a35406188e2582 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.025079 18502 tablet_replica.cc:333] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:58.025336 18502 raft_consensus.cc:2243] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.025578 18502 raft_consensus.cc:2272] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.026798 18502 ts_tablet_manager.cc:1916] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.032732 18373 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.032943 18373 ts_tablet_manager.cc:1403] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.164s user 0.017s sys 0.036s
I20260812 20:03:58.034092 18373 raft_consensus.cc:359] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.034334 18373 raft_consensus.cc:385] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.034454 18373 raft_consensus.cc:740] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.034750 18373 consensus_queue.cc:260] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.035547 18501 ts_tablet_manager.cc:1950] T 5637eaa46a444685a84db3a4a50c0e3b P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.036717 18503 raft_consensus.cc:2243] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.036911 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5637eaa46a444685a84db3a4a50c0e3b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.037065 18503 raft_consensus.cc:2272] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.037479 18501 tablet_replica.cc:333] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.037865 18502 ts_tablet_manager.cc:1929] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.038204 18501 raft_consensus.cc:2243] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.038245 18373 ts_tablet_manager.cc:1434] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 20:03:58.038733 18373 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.040798 18373 tablet_bootstrap.cc:654] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.042441 18502 log.cc:1199] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/f7c30fa50d4049c2a3370881418edf80
I20260812 20:03:58.042817 18501 raft_consensus.cc:2272] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.043452 18502 ts_tablet_manager.cc:1950] T f7c30fa50d4049c2a3370881418edf80 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:58.037922 18515 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.044775 18508 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.044973 18508 ts_tablet_manager.cc:1403] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20260812 20:03:58.045948 18508 raft_consensus.cc:359] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.046042 18501 ts_tablet_manager.cc:1916] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.046155 18508 raft_consensus.cc:385] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.046244 18508 raft_consensus.cc:740] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.046547 18508 consensus_queue.cc:260] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.051960 18515 tablet_bootstrap.cc:654] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:58.055492 17883 consensus_peers.cc:597] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d50211ef1e684ab4824281af94b9febe. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.055611 18508 ts_tablet_manager.cc:1434] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260812 20:03:58.056540 18508 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.062235 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet f7c30fa50d4049c2a3370881418edf80 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.063146 18508 tablet_bootstrap.cc:654] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.064680 18502 tablet_replica.cc:333] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:58.064924 18502 raft_consensus.cc:2243] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.065236 18503 ts_tablet_manager.cc:1916] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.065292 18373 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.065723 18373 ts_tablet_manager.cc:1403] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.010s
I20260812 20:03:58.065965 18502 raft_consensus.cc:2272] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.067130 18502 ts_tablet_manager.cc:1916] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.067298 18373 raft_consensus.cc:359] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.067620 18373 raft_consensus.cc:385] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.067741 18373 raft_consensus.cc:740] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.068127 18373 consensus_queue.cc:260] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.074079 18373 ts_tablet_manager.cc:1434] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260812 20:03:58.075476 18503 ts_tablet_manager.cc:1929] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.075632 18503 log.cc:1199] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/2f51859318ec4174be3dd534658149c2
I20260812 20:03:58.076103 18503 ts_tablet_manager.cc:1950] T 2f51859318ec4174be3dd534658149c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.077359 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 2f51859318ec4174be3dd534658149c2 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.079779 18373 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.079821 18501 ts_tablet_manager.cc:1929] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.079972 18501 log.cc:1199] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/95f1ea51abb743b7b7b6b5feac8af23b
I20260812 20:03:58.080524 18501 ts_tablet_manager.cc:1950] T 95f1ea51abb743b7b7b6b5feac8af23b P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.081956 18502 ts_tablet_manager.cc:1929] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.082123 18502 log.cc:1199] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/07fbf74c9ba7470ea946e2b437b3d71e
I20260812 20:03:58.082710 18502 ts_tablet_manager.cc:1950] T 07fbf74c9ba7470ea946e2b437b3d71e P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:58.083860 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 07fbf74c9ba7470ea946e2b437b3d71e (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.084430 18515 tablet_bootstrap.cc:492] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.084640 18515 ts_tablet_manager.cc:1403] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.006s
I20260812 20:03:58.085108 18502 tablet_replica.cc:333] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:58.085316 18502 raft_consensus.cc:2243] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.085497 18502 raft_consensus.cc:2272] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.085760 18515 raft_consensus.cc:359] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:58.086010 18515 raft_consensus.cc:385] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.086099 18515 raft_consensus.cc:740] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.087574 18508 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.087864 18508 ts_tablet_manager.cc:1403] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20260812 20:03:58.088368 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.373s user 0.062s sys 0.051s Metrics: {"bytes_written":18500,"cfile_cache_hit":35,"cfile_cache_hit_bytes":11057,"cfile_cache_miss":51,"cfile_cache_miss_bytes":15005,"cfile_init":14,"delta_iterators_relevant":19,"dirs.queue_time_us":3355,"dirs.run_cpu_time_us":2252,"dirs.run_wall_time_us":12880,"drs_written":1,"lbm_read_time_us":2243,"lbm_reads_lt_1ms":107,"lbm_write_time_us":1046,"lbm_writes_lt_1ms":27,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":37,"thread_start_us":317215,"threads_started":4}
I20260812 20:03:58.089005 18508 raft_consensus.cc:359] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.089320 18508 raft_consensus.cc:385] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.089478 18508 raft_consensus.cc:740] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.089870 18508 consensus_queue.cc:260] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.090301 18502 ts_tablet_manager.cc:1916] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.091198 18503 tablet_replica.cc:333] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.091441 18503 raft_consensus.cc:2243] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.091792 18503 raft_consensus.cc:2272] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.092912 18503 ts_tablet_manager.cc:1916] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.093532 18508 ts_tablet_manager.cc:1434] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 20:03:58.094024 18508 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.097371 18502 ts_tablet_manager.cc:1929] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.097535 18502 log.cc:1199] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/f61d5f81507649418fcd65eee57dc3f3
I20260812 20:03:58.097986 18515 consensus_queue.cc:260] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:58.100246 18515 ts_tablet_manager.cc:1434] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20260812 20:03:58.100608 18515 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.097443 18508 tablet_bootstrap.cc:654] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.105346 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 95f1ea51abb743b7b7b6b5feac8af23b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.107275 18515 tablet_bootstrap.cc:654] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.109910 18373 tablet_bootstrap.cc:654] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.110663 18508 tablet_bootstrap.cc:492] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.110854 18508 ts_tablet_manager.cc:1403] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260812 20:03:58.111034 18502 ts_tablet_manager.cc:1950] T f61d5f81507649418fcd65eee57dc3f3 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:58.111202 18503 ts_tablet_manager.cc:1929] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.111382 18503 log.cc:1199] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/e3950e2b65124f5aa883e4e872b8777f
I20260812 20:03:58.111760 18508 raft_consensus.cc:359] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.111908 18503 ts_tablet_manager.cc:1950] T e3950e2b65124f5aa883e4e872b8777f P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.111984 18508 raft_consensus.cc:385] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.112088 18508 raft_consensus.cc:740] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.112380 18508 consensus_queue.cc:260] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.113121 18502 tablet_replica.cc:333] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:58.113394 18502 raft_consensus.cc:2243] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.113466 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet e3950e2b65124f5aa883e4e872b8777f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.113659 18508 ts_tablet_manager.cc:1434] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 20:03:58.107787 18501 tablet_replica.cc:333] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.113852 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet f61d5f81507649418fcd65eee57dc3f3 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.114001 18508 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.114146 18501 raft_consensus.cc:2243] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.114217 18502 raft_consensus.cc:2272] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.115388 18502 ts_tablet_manager.cc:1916] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.116164 18501 raft_consensus.cc:2272] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.116701 18508 tablet_bootstrap.cc:654] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.117291 18501 ts_tablet_manager.cc:1916] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.121924 18503 tablet_replica.cc:333] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.122195 18503 raft_consensus.cc:2243] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.122690 18503 raft_consensus.cc:2272] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.131949 18503 ts_tablet_manager.cc:1916] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.133738 18373 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.133948 18373 ts_tablet_manager.cc:1403] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.007s
I20260812 20:03:58.134087 18508 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.134272 18508 ts_tablet_manager.cc:1403] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.002s
I20260812 20:03:58.134994 18508 raft_consensus.cc:359] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.134974 18373 raft_consensus.cc:359] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.135211 18508 raft_consensus.cc:385] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.135200 18373 raft_consensus.cc:385] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.135282 18508 raft_consensus.cc:740] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.135313 18373 raft_consensus.cc:740] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.135637 18373 consensus_queue.cc:260] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.136691 18502 ts_tablet_manager.cc:1929] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.136911 18502 log.cc:1199] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/ba44ce3aac624af697574a4274f38bf5
I20260812 20:03:58.137553 18502 ts_tablet_manager.cc:1950] T ba44ce3aac624af697574a4274f38bf5 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:58.138955 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet ba44ce3aac624af697574a4274f38bf5 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.139470 18502 tablet_replica.cc:333] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:58.139773 18502 raft_consensus.cc:2243] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.140043 18502 raft_consensus.cc:2272] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.141534 18502 ts_tablet_manager.cc:1916] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.141781 18373 ts_tablet_manager.cc:1434] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260812 20:03:58.141897 18508 consensus_queue.cc:260] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.146123 18373 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.147409 18508 ts_tablet_manager.cc:1434] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260812 20:03:58.148260 18373 tablet_bootstrap.cc:654] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.148451 18508 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.150372 18508 tablet_bootstrap.cc:654] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.151422 18515 tablet_bootstrap.cc:492] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.151623 18515 ts_tablet_manager.cc:1403] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20260812 20:03:58.152338 18502 ts_tablet_manager.cc:1929] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.152580 18502 log.cc:1199] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/c71fd265588a4fccae425a41ef98fb05
I20260812 20:03:58.152588 18515 raft_consensus.cc:359] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.152787 18515 raft_consensus.cc:385] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.152894 18515 raft_consensus.cc:740] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.153203 18515 consensus_queue.cc:260] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.154026 18515 ts_tablet_manager.cc:1434] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:58.154625 18515 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.156970 18515 tablet_bootstrap.cc:654] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.153532 18502 ts_tablet_manager.cc:1950] T c71fd265588a4fccae425a41ef98fb05 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:58.161449 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet c71fd265588a4fccae425a41ef98fb05 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.164119 18508 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.164295 18508 ts_tablet_manager.cc:1403] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260812 20:03:58.165074 18501 ts_tablet_manager.cc:1929] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.165277 18501 log.cc:1199] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/2f51859318ec4174be3dd534658149c2
I20260812 20:03:58.165179 18508 raft_consensus.cc:359] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.165441 18508 raft_consensus.cc:385] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.165527 18508 raft_consensus.cc:740] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.165783 18508 consensus_queue.cc:260] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.166009 18501 ts_tablet_manager.cc:1950] T 2f51859318ec4174be3dd534658149c2 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.166340 18508 ts_tablet_manager.cc:1434] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:58.166651 18508 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.166837 18373 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.167081 18373 ts_tablet_manager.cc:1403] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20260812 20:03:58.167587 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 2f51859318ec4174be3dd534658149c2 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.168067 18501 tablet_replica.cc:333] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.168059 18373 raft_consensus.cc:359] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.168300 18373 raft_consensus.cc:385] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.168386 18373 raft_consensus.cc:740] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.168570 18501 raft_consensus.cc:2243] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.168651 18373 consensus_queue.cc:260] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:58.169014 17883 consensus_peers.cc:597] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61d5f81507649418fcd65eee57dc3f3. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.169190 18373 ts_tablet_manager.cc:1434] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:58.169508 18373 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.170063 18508 tablet_bootstrap.cc:654] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.171377 18373 tablet_bootstrap.cc:654] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.178045 18501 raft_consensus.cc:2272] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.180371 18503 ts_tablet_manager.cc:1929] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.177737 18515 tablet_bootstrap.cc:492] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.180924 18515 ts_tablet_manager.cc:1403] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20260812 20:03:58.181072 18373 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.181237 18373 ts_tablet_manager.cc:1403] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 20:03:58.182082 18515 raft_consensus.cc:359] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.182117 18373 raft_consensus.cc:359] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.182312 18373 raft_consensus.cc:385] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.182417 18373 raft_consensus.cc:740] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.182469 18515 raft_consensus.cc:385] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.182741 18515 raft_consensus.cc:740] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.183089 18515 consensus_queue.cc:260] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.183145 18373 consensus_queue.cc:260] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.183734 18373 ts_tablet_manager.cc:1434] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:58.184070 18373 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.186502 18373 tablet_bootstrap.cc:654] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.191811 18515 ts_tablet_manager.cc:1434] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260812 20:03:58.193425 18515 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.193034 18501 ts_tablet_manager.cc:1916] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.195724 18515 tablet_bootstrap.cc:654] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.197335 18503 log.cc:1199] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/95f1ea51abb743b7b7b6b5feac8af23b
I20260812 20:03:58.197844 18503 ts_tablet_manager.cc:1950] T 95f1ea51abb743b7b7b6b5feac8af23b P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.199081 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 95f1ea51abb743b7b7b6b5feac8af23b (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.197031 18508 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.199466 18508 ts_tablet_manager.cc:1403] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.011s
I20260812 20:03:58.200305 18508 raft_consensus.cc:359] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.200538 18508 raft_consensus.cc:385] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.200618 18508 raft_consensus.cc:740] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.200897 18508 consensus_queue.cc:260] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.201273 18503 tablet_replica.cc:333] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.201511 18503 raft_consensus.cc:2243] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.201516 18481 raft_consensus.cc:493] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.201929 18481 raft_consensus.cc:515] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:58.202879 18481 leader_election.cc:290] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:03:58.203161 18508 ts_tablet_manager.cc:1434] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260812 20:03:58.203486 18508 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.205497 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2950ec0320c432887522c75de50d8c5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.205585 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2950ec0320c432887522c75de50d8c5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.205772 18373 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.205772 18277 raft_consensus.cc:2468] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.205837 17962 raft_consensus.cc:2468] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.206000 18373 ts_tablet_manager.cc:1403] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.001s
I20260812 20:03:58.206317 18043 leader_election.cc:304] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.206881 18481 raft_consensus.cc:2804] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.206983 18481 raft_consensus.cc:493] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.206916 18373 raft_consensus.cc:359] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.207084 18481 raft_consensus.cc:3060] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.207130 18373 raft_consensus.cc:385] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.207211 18373 raft_consensus.cc:740] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.207875 18373 consensus_queue.cc:260] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.208459 18373 ts_tablet_manager.cc:1434] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:03:58.208838 18373 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.210104 18481 raft_consensus.cc:515] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:58.210815 18373 tablet_bootstrap.cc:654] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.211658 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2950ec0320c432887522c75de50d8c5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.211862 17962 raft_consensus.cc:3060] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.212831 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2950ec0320c432887522c75de50d8c5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.213069 18277 raft_consensus.cc:3060] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.215138 17962 raft_consensus.cc:2468] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.215830 18043 leader_election.cc:304] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.216145 18377 raft_consensus.cc:2804] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.216338 18377 raft_consensus.cc:697] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:58.216702 18503 raft_consensus.cc:2272] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.216637 18377 consensus_queue.cc:237] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:03:58.217648 18508 tablet_bootstrap.cc:654] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.219858 18277 raft_consensus.cc:2468] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.221315 18373 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.221345 18481 leader_election.cc:290] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:03:58.221480 18373 ts_tablet_manager.cc:1403] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260812 20:03:58.222453 18373 raft_consensus.cc:359] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.222671 18373 raft_consensus.cc:385] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.222796 18373 raft_consensus.cc:740] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.223096 18373 consensus_queue.cc:260] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.223769 18373 ts_tablet_manager.cc:1434] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:03:58.224105 18373 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.225932 18503 ts_tablet_manager.cc:1916] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.228164 17767 catalog_manager.cc:5719] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.229127 18406 raft_consensus.cc:493] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.229285 18406 raft_consensus.cc:515] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.229952 18406 leader_election.cc:290] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.230341 18373 tablet_bootstrap.cc:654] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.231145 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e5c8e0a860641168abfaee4c7216fd7" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.231364 17962 raft_consensus.cc:2468] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.232200 18043 leader_election.cc:304] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.232570 18406 raft_consensus.cc:2804] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.232693 18406 raft_consensus.cc:493] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.232798 18406 raft_consensus.cc:3060] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.232721 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e5c8e0a860641168abfaee4c7216fd7" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.232941 18279 raft_consensus.cc:2468] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
W20260812 20:03:58.234226 18043 consensus_peers.cc:597] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07fbf74c9ba7470ea946e2b437b3d71e. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.236438 18501 ts_tablet_manager.cc:1929] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.236640 18501 log.cc:1199] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/e3950e2b65124f5aa883e4e872b8777f
I20260812 20:03:58.236582 18406 raft_consensus.cc:515] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.237217 18501 ts_tablet_manager.cc:1950] T e3950e2b65124f5aa883e4e872b8777f P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.238062 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e5c8e0a860641168abfaee4c7216fd7" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.238327 17962 raft_consensus.cc:3060] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.238513 18501 tablet_replica.cc:333] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.238796 18501 raft_consensus.cc:2243] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.238991 18501 raft_consensus.cc:2272] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.242453 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet e3950e2b65124f5aa883e4e872b8777f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.244503 17962 raft_consensus.cc:2468] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.244781 18508 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.244984 18508 ts_tablet_manager.cc:1403] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.002s
I20260812 20:03:58.245461 18043 leader_election.cc:304] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.245922 18481 raft_consensus.cc:2804] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.245926 18508 raft_consensus.cc:359] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.246124 18508 raft_consensus.cc:385] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.246147 18481 raft_consensus.cc:697] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:58.246248 18508 raft_consensus.cc:740] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.246491 18508 consensus_queue.cc:260] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.246503 18481 consensus_queue.cc:237] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.247110 18508 ts_tablet_manager.cc:1434] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:58.247478 18508 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.249441 18508 tablet_bootstrap.cc:654] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.252895 18406 leader_election.cc:290] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.253034 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e5c8e0a860641168abfaee4c7216fd7" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.253319 18277 raft_consensus.cc:3060] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.254109 18501 ts_tablet_manager.cc:1916] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.256686 18277 raft_consensus.cc:2468] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.269990 18501 ts_tablet_manager.cc:1929] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.270190 18501 log.cc:1199] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5a00feea243f440d9348756186c6df32
I20260812 20:03:58.270825 18501 ts_tablet_manager.cc:1950] T 5a00feea243f440d9348756186c6df32 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.272267 18501 tablet_replica.cc:333] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.272511 18501 raft_consensus.cc:2243] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.272686 18515 tablet_bootstrap.cc:492] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.272835 18515 ts_tablet_manager.cc:1403] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.017s
I20260812 20:03:58.272956 18501 raft_consensus.cc:2272] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.273801 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5a00feea243f440d9348756186c6df32 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.274261 18515 raft_consensus.cc:359] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.274531 18515 raft_consensus.cc:385] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.274636 18515 raft_consensus.cc:740] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.275050 18515 consensus_queue.cc:260] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.275769 18515 ts_tablet_manager.cc:1434] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:58.276202 18515 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.279007 18471 raft_consensus.cc:493] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.279222 18471 raft_consensus.cc:515] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.280215 18373 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.280288 18471 leader_election.cc:290] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.280485 18373 ts_tablet_manager.cc:1403] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.002s
I20260812 20:03:58.281364 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c9d056274f4c1ab24c5f317981d016" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.281548 18373 raft_consensus.cc:359] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.282048 18373 raft_consensus.cc:385] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.282158 18373 raft_consensus.cc:740] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
W20260812 20:03:58.282186 17884 leader_election.cc:343] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.282519 18373 consensus_queue.cc:260] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.282970 18515 tablet_bootstrap.cc:654] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.283406 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c9d056274f4c1ab24c5f317981d016" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.283641 18277 raft_consensus.cc:2468] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.284318 17883 leader_election.cc:304] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:58.284686 18501 ts_tablet_manager.cc:1916] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.285162 18471 raft_consensus.cc:2804] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.285293 18471 raft_consensus.cc:493] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.285389 18471 raft_consensus.cc:3060] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.285833 18373 ts_tablet_manager.cc:1434] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 20:03:58.286280 18373 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.288638 18471 raft_consensus.cc:515] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.288836 18373 tablet_bootstrap.cc:654] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.290087 18406 raft_consensus.cc:493] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.290266 18406 raft_consensus.cc:515] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.290386 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c9d056274f4c1ab24c5f317981d016" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.291100 18406 leader_election.cc:290] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
W20260812 20:03:58.291971 17884 leader_election.cc:343] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.292078 18501 ts_tablet_manager.cc:1929] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.292205 18501 log.cc:1199] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b8f649dddfa5466988e196d03656eb6f
I20260812 20:03:58.292448 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.292709 18501 ts_tablet_manager.cc:1950] T b8f649dddfa5466988e196d03656eb6f P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.292711 18277 raft_consensus.cc:2468] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.293542 18043 leader_election.cc:304] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.293871 18406 raft_consensus.cc:2804] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.294009 18406 raft_consensus.cc:493] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.293960 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b8f649dddfa5466988e196d03656eb6f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.294118 18406 raft_consensus.cc:3060] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.294256 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.294665 17962 raft_consensus.cc:2468] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.296564 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c9d056274f4c1ab24c5f317981d016" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.296830 18277 raft_consensus.cc:3060] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.297412 18406 raft_consensus.cc:515] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.299016 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.299289 17962 raft_consensus.cc:3060] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.300846 18277 raft_consensus.cc:2468] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
W20260812 20:03:58.301147 17751 catalog_manager.cc:5017] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): delete failed for tablet ecfbe43f35904b94b26f8e6a239cf393 because the tablet was not found. No further retry: Not found: Tablet not found: ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:58.301658 17883 leader_election.cc:304] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:58.301899 18501 tablet_replica.cc:333] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.302039 18379 raft_consensus.cc:2804] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.302503 18471 leader_election.cc:290] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.302711 18379 raft_consensus.cc:697] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.303056 18379 consensus_queue.cc:237] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.304140 18406 leader_election.cc:290] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.305035 17962 raft_consensus.cc:2468] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.305567 18043 leader_election.cc:304] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.305927 18406 raft_consensus.cc:2804] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.306100 18406 raft_consensus.cc:697] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:58.306391 18406 consensus_queue.cc:237] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:58.314831 17883 consensus_peers.cc:597] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7c30fa50d4049c2a3370881418edf80. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.316516 17768 catalog_manager.cc:5719] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.317307 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad9b9e5ec944a1aa309cc9332b2dbca" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.317380 18503 ts_tablet_manager.cc:1929] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.317447 18501 raft_consensus.cc:2243] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.317541 18503 log.cc:1199] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/11d7acaf0d9645ada0116c85aaac2c40
I20260812 20:03:58.317713 18501 raft_consensus.cc:2272] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.317770 18277 raft_consensus.cc:3060] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.318758 18501 ts_tablet_manager.cc:1916] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.319770 18503 ts_tablet_manager.cc:1950] T 11d7acaf0d9645ada0116c85aaac2c40 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.321213 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 11d7acaf0d9645ada0116c85aaac2c40 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.321765 18277 raft_consensus.cc:2468] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.322613 18503 tablet_replica.cc:333] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.323561 18503 raft_consensus.cc:2243] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.324033 18503 raft_consensus.cc:2272] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.324951 18508 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.325155 18508 ts_tablet_manager.cc:1403] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.032s
I20260812 20:03:58.326222 18508 raft_consensus.cc:359] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.326459 18508 raft_consensus.cc:385] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.326556 18508 raft_consensus.cc:740] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.326572 18373 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.326759 18373 ts_tablet_manager.cc:1403] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.005s
I20260812 20:03:58.326916 18508 consensus_queue.cc:260] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.327663 18508 ts_tablet_manager.cc:1434] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:58.327651 18373 raft_consensus.cc:359] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.327845 18373 raft_consensus.cc:385] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.327947 18373 raft_consensus.cc:740] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.328039 18508 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.328286 18373 consensus_queue.cc:260] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.329108 18501 ts_tablet_manager.cc:1929] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.329308 18501 log.cc:1199] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/d50211ef1e684ab4824281af94b9febe
I20260812 20:03:58.329834 18501 ts_tablet_manager.cc:1950] T d50211ef1e684ab4824281af94b9febe P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.330000 18508 tablet_bootstrap.cc:654] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.330092 18373 ts_tablet_manager.cc:1434] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:03:58.330441 18373 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.331033 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet d50211ef1e684ab4824281af94b9febe (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.331923 18515 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.331992 18503 ts_tablet_manager.cc:1916] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.332108 18515 ts_tablet_manager.cc:1403] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.011s
I20260812 20:03:58.332295 18373 tablet_bootstrap.cc:654] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.333087 18515 raft_consensus.cc:359] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.333309 18515 raft_consensus.cc:385] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.333401 18515 raft_consensus.cc:740] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.333897 18515 consensus_queue.cc:260] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.334474 18515 ts_tablet_manager.cc:1434] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:58.334795 18515 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.336673 18515 tablet_bootstrap.cc:654] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.337913 18501 tablet_replica.cc:333] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.338121 18501 raft_consensus.cc:2243] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.338513 18501 raft_consensus.cc:2272] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.340716 18503 ts_tablet_manager.cc:1929] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.340891 18503 log.cc:1199] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5a00feea243f440d9348756186c6df32
I20260812 20:03:58.340973 18501 ts_tablet_manager.cc:1916] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.341423 18503 ts_tablet_manager.cc:1950] T 5a00feea243f440d9348756186c6df32 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.341992 17767 catalog_manager.cc:5719] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.342710 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5a00feea243f440d9348756186c6df32 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.344291 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.344683 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.345049 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.344910 17767 catalog_manager.cc:5719] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.345328 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.345857 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.346205 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.346434 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.347433 18515 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.347609 18515 ts_tablet_manager.cc:1403] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20260812 20:03:58.348495 18515 raft_consensus.cc:359] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.348683 18515 raft_consensus.cc:385] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.348771 18515 raft_consensus.cc:740] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.349051 18515 consensus_queue.cc:260] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.349203 18501 ts_tablet_manager.cc:1929] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.349431 18501 log.cc:1199] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/530fa3bb1bdf4fa083a35406188e2582
I20260812 20:03:58.349761 18515 ts_tablet_manager.cc:1434] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:58.349992 18501 ts_tablet_manager.cc:1950] T 530fa3bb1bdf4fa083a35406188e2582 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.350407 18515 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.351652 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 530fa3bb1bdf4fa083a35406188e2582 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.352041 18501 tablet_replica.cc:333] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.352254 18501 raft_consensus.cc:2243] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.352514 18501 raft_consensus.cc:2272] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.352588 18515 tablet_bootstrap.cc:654] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.352479 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":171,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.353516 18501 ts_tablet_manager.cc:1916] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.354386 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.354761 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.355082 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.355417 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.357573 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.357930 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.358197 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.359778 18503 tablet_replica.cc:333] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.360117 18503 raft_consensus.cc:2243] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.360426 18503 raft_consensus.cc:2272] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.361549 18503 ts_tablet_manager.cc:1916] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.364148 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.364540 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.364879 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.365195 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.367587 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.367952 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.368228 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.368623 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":189,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":8983424,"update_count":1}
I20260812 20:03:58.368983 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.369308 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.369743 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.370182 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.370458 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.373921 18501 ts_tablet_manager.cc:1929] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.374178 18501 log.cc:1199] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/f7c30fa50d4049c2a3370881418edf80
I20260812 20:03:58.374899 18501 ts_tablet_manager.cc:1950] T f7c30fa50d4049c2a3370881418edf80 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.375082 18379 raft_consensus.cc:493] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.375319 18379 raft_consensus.cc:515] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.376379 18379 leader_election.cc:290] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.376678 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet f7c30fa50d4049c2a3370881418edf80 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.377328 18501 tablet_replica.cc:333] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.377516 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb77a5184b8a459db4499e77cd2df733" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.377699 18501 raft_consensus.cc:2243] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.377907 18126 raft_consensus.cc:2468] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.378333 18501 raft_consensus.cc:2272] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.378885 17884 leader_election.cc:304] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.379959 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.380388 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.380718 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.381076 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.381801 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.381846 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb77a5184b8a459db4499e77cd2df733" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.382159 18277 raft_consensus.cc:2468] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.382197 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.382452 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.383208 18379 raft_consensus.cc:2804] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.383378 18379 raft_consensus.cc:493] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.383489 18379 raft_consensus.cc:3060] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.383611 18373 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.383844 18373 ts_tablet_manager.cc:1403] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.002s
I20260812 20:03:58.384980 18373 raft_consensus.cc:359] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.385242 18373 raft_consensus.cc:385] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.385332 18373 raft_consensus.cc:740] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.388422 18379 raft_consensus.cc:515] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.390501 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb77a5184b8a459db4499e77cd2df733" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.390823 18126 raft_consensus.cc:3060] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.394235 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.394681 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.395005 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.395126 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb77a5184b8a459db4499e77cd2df733" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.395430 18277 raft_consensus.cc:3060] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.395481 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.395835 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.399694 18443 raft_consensus.cc:493] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.399910 18443 raft_consensus.cc:515] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.400388 18277 raft_consensus.cc:2468] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.400781 18443 leader_election.cc:290] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.401068 17883 leader_election.cc:304] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.401831 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.401974 18421 raft_consensus.cc:2804] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.402148 18421 raft_consensus.cc:697] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.402482 18421 consensus_queue.cc:237] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:58.402693 18208 leader_election.cc:343] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.403467 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:03:58.404208 18207 leader_election.cc:343] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.404354 18207 leader_election.cc:304] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:58.404726 18373 consensus_queue.cc:260] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.404949 18443 raft_consensus.cc:2749] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:58.405966 18126 raft_consensus.cc:2468] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.416673 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.003s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":264,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.419255 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.054s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":199,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.421428 18379 leader_election.cc:290] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.422176 18501 ts_tablet_manager.cc:1916] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.429554 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":8158,"lbm_read_time_us":201,"lbm_reads_lt_1ms":8,"lbm_write_time_us":250,"lbm_writes_lt_1ms":4,"mutex_wait_us":6126,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.427148 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.057s user 0.006s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2910,"lbm_read_time_us":177,"lbm_reads_lt_1ms":8,"lbm_write_time_us":225,"lbm_writes_lt_1ms":4,"mutex_wait_us":38116,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.433821 18373 ts_tablet_manager.cc:1434] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.050s user 0.005s sys 0.004s
I20260812 20:03:58.434451 18373 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.437259 18373 tablet_bootstrap.cc:654] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.439126 17768 catalog_manager.cc:5719] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.442942 18503 ts_tablet_manager.cc:1929] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.443917 18508 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.444329 18508 ts_tablet_manager.cc:1403] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.000s
I20260812 20:03:58.445333 18508 raft_consensus.cc:359] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.445528 18508 raft_consensus.cc:385] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.445639 18508 raft_consensus.cc:740] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.445892 18508 consensus_queue.cc:260] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.446486 18508 ts_tablet_manager.cc:1434] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:58.446871 18508 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.449812 18567 raft_consensus.cc:493] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.450098 18567 raft_consensus.cc:515] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.451027 18508 tablet_bootstrap.cc:654] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.452144 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.452570 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.452889 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.452980 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "489ea23ba78a47bda493198355f9b340" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.453186 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.453152 18567 leader_election.cc:290] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.453239 17962 raft_consensus.cc:2468] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.454836 18207 leader_election.cc:304] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.455317 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.455622 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.455642 18567 raft_consensus.cc:2804] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.455806 18567 raft_consensus.cc:493] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.455914 18567 raft_consensus.cc:3060] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.455921 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.456333 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "489ea23ba78a47bda493198355f9b340" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.456636 18126 raft_consensus.cc:2468] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.458832 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.459177 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.459467 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.459990 18421 raft_consensus.cc:493] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.460163 18421 raft_consensus.cc:515] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.460891 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.461176 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.461575 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.461869 18126 raft_consensus.cc:2468] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.462517 17884 leader_election.cc:304] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.462868 18421 leader_election.cc:290] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.463472 18567 raft_consensus.cc:515] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.463573 18421 raft_consensus.cc:2804] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.463732 18421 raft_consensus.cc:493] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.463840 18421 raft_consensus.cc:3060] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.464849 18373 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.465014 18373 ts_tablet_manager.cc:1403] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20260812 20:03:58.465632 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.465989 18373 raft_consensus.cc:359] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.466177 18373 raft_consensus.cc:385] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.466288 18373 raft_consensus.cc:740] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.466418 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "489ea23ba78a47bda493198355f9b340" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.466542 18373 consensus_queue.cc:260] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.466665 18126 raft_consensus.cc:3060] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.466615 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "489ea23ba78a47bda493198355f9b340" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.466902 17962 raft_consensus.cc:3060] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.467171 18373 ts_tablet_manager.cc:1434] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:03:58.467510 18373 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.468263 18421 raft_consensus.cc:515] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.469456 18373 tablet_bootstrap.cc:654] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:58.469890 17883 leader_election.cc:343] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.470737 18126 raft_consensus.cc:2468] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.471477 18208 leader_election.cc:304] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.471807 18443 raft_consensus.cc:2804] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.471975 18567 leader_election.cc:290] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.473253 18501 ts_tablet_manager.cc:1929] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.473413 18501 log.cc:1199] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/07fbf74c9ba7470ea946e2b437b3d71e
I20260812 20:03:58.473974 18501 ts_tablet_manager.cc:1950] T 07fbf74c9ba7470ea946e2b437b3d71e P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.475455 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 07fbf74c9ba7470ea946e2b437b3d71e (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.478165 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.478808 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.480062 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.480310 18126 raft_consensus.cc:3060] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.485201 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.485648 18126 raft_consensus.cc:2468] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.486196 17884 leader_election.cc:304] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.486685 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.487035 18379 raft_consensus.cc:2804] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.487253 18379 raft_consensus.cc:697] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.487553 17962 raft_consensus.cc:2468] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.487600 18379 consensus_queue.cc:237] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.489269 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.036s user 0.005s sys 0.000s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.489952 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
W20260812 20:03:58.491925 17884 consensus_peers.cc:597] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d50211ef1e684ab4824281af94b9febe. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.493727 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.494050 18503 log.cc:1199] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b8f649dddfa5466988e196d03656eb6f
I20260812 20:03:58.494114 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.494395 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.494626 18503 ts_tablet_manager.cc:1950] T b8f649dddfa5466988e196d03656eb6f P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.494686 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.496949 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.497313 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.497534 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.498436 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b8f649dddfa5466988e196d03656eb6f (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.498798 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.499163 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.499432 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.173172
I20260812 20:03:58.499845 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.499955 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.500098 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260812 20:03:58.500846 17883 leader_election.cc:343] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:58.501320 17750 catalog_manager.cc:5017] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): delete failed for tablet ecfbe43f35904b94b26f8e6a239cf393 because the tablet was not found. No further retry: Not found: Tablet not found: ecfbe43f35904b94b26f8e6a239cf393
I20260812 20:03:58.502947 18421 leader_election.cc:290] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.504674 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.504976 18503 tablet_replica.cc:333] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.505247 18503 raft_consensus.cc:2243] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.505246 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.505739 18503 raft_consensus.cc:2272] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.506683 18503 ts_tablet_manager.cc:1916] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.507037 18567 raft_consensus.cc:493] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.512566 18515 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.513159 18515 ts_tablet_manager.cc:1403] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.163s user 0.053s sys 0.076s
I20260812 20:03:58.513886 18508 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.514086 18508 ts_tablet_manager.cc:1403] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.006s
I20260812 20:03:58.514186 18515 raft_consensus.cc:359] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.514436 18515 raft_consensus.cc:385] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.514526 18515 raft_consensus.cc:740] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.514801 18515 consensus_queue.cc:260] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.514990 18508 raft_consensus.cc:359] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.515242 18508 raft_consensus.cc:385] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.515326 18508 raft_consensus.cc:740] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.515349 18515 ts_tablet_manager.cc:1434] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:03:58.515578 18508 consensus_queue.cc:260] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.515705 18515 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.517598 18515 tablet_bootstrap.cc:654] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.518546 17768 catalog_manager.cc:5719] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.519881 18508 ts_tablet_manager.cc:1434] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260812 20:03:58.520252 18508 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.514335 18567 raft_consensus.cc:515] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.521845 18515 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.522022 18515 ts_tablet_manager.cc:1403] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260812 20:03:58.522428 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.522452 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.522895 18515 raft_consensus.cc:359] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.523064 18515 raft_consensus.cc:385] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260812 20:03:58.523075 18208 leader_election.cc:343] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.523161 18515 raft_consensus.cc:740] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
W20260812 20:03:58.523272 18207 leader_election.cc:343] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.523401 18207 leader_election.cc:304] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:58.523418 18515 consensus_queue.cc:260] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.523870 18570 raft_consensus.cc:2749] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:58.523960 18515 ts_tablet_manager.cc:1434] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:58.524288 18515 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.524137 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.007s sys 0.000s Metrics: {"bytes_written":481,"delete_count":0,"lbm_write_time_us":192,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260812 20:03:58.526662 18373 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.526880 18373 ts_tablet_manager.cc:1403] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.011s
I20260812 20:03:58.523985 18567 leader_election.cc:290] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.527805 18373 raft_consensus.cc:359] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.528015 18373 raft_consensus.cc:385] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.528096 18373 raft_consensus.cc:740] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.528337 18373 consensus_queue.cc:260] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.528903 18373 ts_tablet_manager.cc:1434] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:58.529255 18373 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.529249 18501 tablet_replica.cc:333] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.529489 18501 raft_consensus.cc:2243] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.529716 18501 raft_consensus.cc:2272] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.531020 18373 tablet_bootstrap.cc:654] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.532155 18515 tablet_bootstrap.cc:654] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.533505 18501 ts_tablet_manager.cc:1916] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.534104 18503 ts_tablet_manager.cc:1929] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.534825 18503 log.cc:1199] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/d50211ef1e684ab4824281af94b9febe
I20260812 20:03:58.535195 18508 tablet_bootstrap.cc:654] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.509348 18443 raft_consensus.cc:697] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:58.536779 18443 consensus_queue.cc:237] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.545991 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.546365 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.546674 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.546891 18503 ts_tablet_manager.cc:1950] T d50211ef1e684ab4824281af94b9febe P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.548589 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet d50211ef1e684ab4824281af94b9febe (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.545552 17768 catalog_manager.cc:5719] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:58.551939 18379 raft_consensus.cc:493] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.552212 18379 raft_consensus.cc:515] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.553117 18379 leader_election.cc:290] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.554149 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.555900 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.556161 18277 raft_consensus.cc:2468] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
W20260812 20:03:58.557075 17884 leader_election.cc:343] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.560508 18373 tablet_bootstrap.cc:492] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.560654 18373 ts_tablet_manager.cc:1403] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.005s
I20260812 20:03:58.561462 18373 raft_consensus.cc:359] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.561672 18373 raft_consensus.cc:385] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.561771 18373 raft_consensus.cc:740] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.562047 18373 consensus_queue.cc:260] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.562326 18567 raft_consensus.cc:493] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.562482 18567 raft_consensus.cc:515] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.562670 18373 ts_tablet_manager.cc:1434] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:58.562970 18373 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.563664 18567 leader_election.cc:290] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.563696 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44959dd27d9642f0ad8ddf8395887083" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:03:58.564493 18207 leader_election.cc:343] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.550312 18503 tablet_replica.cc:333] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.565577 18503 raft_consensus.cc:2243] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.566988 18503 raft_consensus.cc:2272] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.567847 18503 ts_tablet_manager.cc:1916] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.568961 18373 tablet_bootstrap.cc:654] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.571013 17883 leader_election.cc:304] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:58.571890 18421 raft_consensus.cc:2804] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.572021 18421 raft_consensus.cc:493] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.572160 18421 raft_consensus.cc:3060] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.574221 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.027s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":27065}
I20260812 20:03:58.575083 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":19624}
I20260812 20:03:58.575572 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.006s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"mutex_wait_us":19358,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.576133 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.076s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12171,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1950,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":4457,"lbm_read_time_us":591,"lbm_reads_lt_1ms":23,"lbm_write_time_us":794,"lbm_writes_lt_1ms":9,"mutex_wait_us":7902,"peak_mem_usage":32768,"reinsert_count":0,"update_count":5}
I20260812 20:03:58.578658 18421 raft_consensus.cc:515] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.579967 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44959dd27d9642f0ad8ddf8395887083" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.580293 18126 raft_consensus.cc:2468] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.581037 18208 leader_election.cc:304] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:58.586216 18567 raft_consensus.cc:2804] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.586489 18567 raft_consensus.cc:493] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.586743 18567 raft_consensus.cc:3060] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.590015 18515 tablet_bootstrap.cc:492] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.589890 18567 raft_consensus.cc:515] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.590209 18515 ts_tablet_manager.cc:1403] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.017s
I20260812 20:03:58.579995 18508 tablet_bootstrap.cc:492] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.590512 18508 ts_tablet_manager.cc:1403] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.004s
I20260812 20:03:58.591101 18515 raft_consensus.cc:359] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.591326 18515 raft_consensus.cc:385] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.591403 18515 raft_consensus.cc:740] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.591427 18508 raft_consensus.cc:359] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.591621 18508 raft_consensus.cc:385] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.591738 18508 raft_consensus.cc:740] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.591688 18515 consensus_queue.cc:260] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.591981 18508 consensus_queue.cc:260] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.592247 18515 ts_tablet_manager.cc:1434] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:58.592545 18508 ts_tablet_manager.cc:1434] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:58.592615 18515 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.592854 18508 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.593961 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.105s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":104984}
I20260812 20:03:58.594322 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.104s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":103975}
I20260812 20:03:58.594699 18515 tablet_bootstrap.cc:654] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.594872 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.108s user 0.005s sys 0.000s Metrics: {"bytes_written":490,"delete_count":0,"lbm_write_time_us":182,"lbm_writes_lt_1ms":4,"mutex_wait_us":100538,"reinsert_count":0,"update_count":3}
I20260812 20:03:58.580125 18503 ts_tablet_manager.cc:1929] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.595547 18503 log.cc:1199] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/530fa3bb1bdf4fa083a35406188e2582
I20260812 20:03:58.596071 18503 ts_tablet_manager.cc:1950] T 530fa3bb1bdf4fa083a35406188e2582 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.596222 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.136s user 0.018s sys 0.014s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12144,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":660,"dirs.run_cpu_time_us":312,"dirs.run_wall_time_us":1330,"lbm_read_time_us":579,"lbm_reads_lt_1ms":23,"lbm_write_time_us":502,"lbm_writes_lt_1ms":9,"mutex_wait_us":11537,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":71182,"threads_started":1,"update_count":4}
I20260812 20:03:58.597860 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 530fa3bb1bdf4fa083a35406188e2582 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.599081 18503 tablet_replica.cc:333] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.599380 18503 raft_consensus.cc:2243] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.599885 18503 raft_consensus.cc:2272] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.600734 18503 ts_tablet_manager.cc:1916] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.604665 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.605026 18508 tablet_bootstrap.cc:654] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.606580 18421 leader_election.cc:290] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.606899 18567 leader_election.cc:290] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.607322 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77032e8e8636495a9dfdfa1ce5d85021" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.607544 18277 raft_consensus.cc:3060] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.607903 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44959dd27d9642f0ad8ddf8395887083" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.608178 17962 raft_consensus.cc:3060] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.611128 18277 raft_consensus.cc:2468] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.612373 17883 leader_election.cc:304] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.612908 18421 raft_consensus.cc:2804] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.613176 18421 raft_consensus.cc:697] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.613744 17962 raft_consensus.cc:2468] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.614130 18421 consensus_queue.cc:237] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.614635 18207 leader_election.cc:304] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.615162 18567 raft_consensus.cc:2804] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.615379 18567 raft_consensus.cc:697] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:58.615783 18567 consensus_queue.cc:237] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.619949 18126 raft_consensus.cc:3060] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.620609 18373 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.620687 18503 ts_tablet_manager.cc:1929] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.620801 18373 ts_tablet_manager.cc:1403] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.017s
I20260812 20:03:58.621789 18373 raft_consensus.cc:359] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.622058 18373 raft_consensus.cc:385] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.622170 18373 raft_consensus.cc:740] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.622457 18373 consensus_queue.cc:260] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.622987 18373 ts_tablet_manager.cc:1434] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 20:03:58.623289 18373 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:58.625181 18373 tablet_bootstrap.cc:654] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.626461 18126 raft_consensus.cc:2468] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.628079 17768 catalog_manager.cc:5719] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
W20260812 20:03:58.628403 17884 consensus_peers.cc:597] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.628682 18406 raft_consensus.cc:493] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.628927 18406 raft_consensus.cc:515] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.629577 18503 log.cc:1199] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/f7c30fa50d4049c2a3370881418edf80
I20260812 20:03:58.629750 18406 leader_election.cc:290] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.630188 18503 ts_tablet_manager.cc:1950] T f7c30fa50d4049c2a3370881418edf80 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.631102 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44302 (request call id 28) took 1029 ms. Trace:
I20260812 20:03:58.631147 18406 consensus_queue.cc:1048] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.631237 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6397453a07df4814be9827bb15738081" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.631443 18277 raft_consensus.cc:2468] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.628652 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44959dd27d9642f0ad8ddf8395887083" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.632601 18379 raft_consensus.cc:493] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.632351 18126 raft_consensus.cc:3060] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.632792 18379 raft_consensus.cc:515] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.633042 18471 raft_consensus.cc:493] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.633275 18508 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.633201 18471 raft_consensus.cc:515] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.633469 18508 ts_tablet_manager.cc:1403] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.007s
I20260812 20:03:58.633587 18379 leader_election.cc:290] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.633924 18471 leader_election.cc:290] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.634467 18508 raft_consensus.cc:359] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.634666 18508 raft_consensus.cc:385] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.634748 18508 raft_consensus.cc:740] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.634976 18508 consensus_queue.cc:260] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.635509 18508 ts_tablet_manager.cc:1434] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:58.635993 18508 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.636492 18126 raft_consensus.cc:2468] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.637183 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6397453a07df4814be9827bb15738081" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.637521 17962 raft_consensus.cc:2468] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:58.638013 18508 tablet_bootstrap.cc:654] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.637975 18501 ts_tablet_manager.cc:1929] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.638226 18501 log.cc:1199] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/f61d5f81507649418fcd65eee57dc3f3
I20260812 20:03:58.638200 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.638453 18279 raft_consensus.cc:2468] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.638975 18501 ts_tablet_manager.cc:1950] T f61d5f81507649418fcd65eee57dc3f3 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.639164 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.639375 18277 raft_consensus.cc:2468] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.639925 17883 leader_election.cc:304] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.631278 18503 rpcz_store.cc:276] 0812 20:03:57.601204 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.601690 (+ 486us) service_pool.cc:224] Handling call
0812 20:03:58.598982 (+997292us) ts_tablet_manager.cc:1151] Deleting tablet f7c30fa50d4049c2a3370881418edf80
0812 20:03:58.598991 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.612702 (+ 13711us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.620535 (+ 7833us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.631055 (+ 10520us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997094}
I20260812 20:03:58.640189 18501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 30) took 1034 ms. Trace:
I20260812 20:03:58.640254 18471 raft_consensus.cc:2804] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.640300 17883 leader_election.cc:304] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.640384 18471 raft_consensus.cc:493] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.640507 18471 raft_consensus.cc:3060] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.640619 18379 raft_consensus.cc:2804] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.640296 18503 tablet_replica.cc:333] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.640941 18503 raft_consensus.cc:2243] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.640332 18501 rpcz_store.cc:276] 0812 20:03:57.605990 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.606043 (+ 53us) service_pool.cc:224] Handling call
0812 20:03:58.529175 (+923132us) ts_tablet_manager.cc:1151] Deleting tablet f61d5f81507649418fcd65eee57dc3f3
0812 20:03:58.529185 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.577611 (+ 48426us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.621376 (+ 43765us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.640122 (+ 18746us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":922895}
I20260812 20:03:58.641350 18503 raft_consensus.cc:2272] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.642264 18503 ts_tablet_manager.cc:1916] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.644907 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.645283 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.645663 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.645977 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.648453 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.648731 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.649010 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.649891 18379 raft_consensus.cc:493] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.650079 18379 raft_consensus.cc:3060] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.650110 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet f61d5f81507649418fcd65eee57dc3f3 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.650568 18501 tablet_replica.cc:333] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.650859 18501 raft_consensus.cc:2243] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.651113 18501 raft_consensus.cc:2272] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.652994 18567 raft_consensus.cc:493] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.653569 18567 raft_consensus.cc:515] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.654901 17767 catalog_manager.cc:5719] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:58.658398 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:58.659089 18208 leader_election.cc:343] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.659423 18373 tablet_bootstrap.cc:492] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:58.659572 18373 ts_tablet_manager.cc:1403] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.010s
I20260812 20:03:58.660583 18373 raft_consensus.cc:359] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.660794 18373 raft_consensus.cc:385] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.660890 18373 raft_consensus.cc:740] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.659379 18567 leader_election.cc:290] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.661928 18373 consensus_queue.cc:260] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.662509 18043 leader_election.cc:304] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.662894 18406 raft_consensus.cc:2804] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.663028 18406 raft_consensus.cc:493] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.663123 18406 raft_consensus.cc:3060] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.663138 18373 ts_tablet_manager.cc:1434] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 20:03:58.663848 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet f7c30fa50d4049c2a3370881418edf80 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.665180 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.665145 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.665488 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.665676 18567 raft_consensus.cc:493] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.665807 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.665854 18567 raft_consensus.cc:515] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.666000 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.666322 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.666410 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.666532 18567 leader_election.cc:290] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.666610 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.666745 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.666971 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.669462 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.669809 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.670077 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.672971 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.673274 18125 raft_consensus.cc:2468] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.675351 18208 leader_election.cc:304] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.675755 18567 raft_consensus.cc:2804] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.675876 18567 raft_consensus.cc:493] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.675989 18567 raft_consensus.cc:3060] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.676425 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.676719 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.676988 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.677294 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.677513 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.679101 18567 raft_consensus.cc:515] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.679837 18567 leader_election.cc:290] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.682559 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.682890 18125 raft_consensus.cc:3060] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.685547 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.685992 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.653600 18379 raft_consensus.cc:515] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.687598 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.687773 18379 leader_election.cc:290] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.666317 18591 raft_consensus.cc:493] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.688006 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.656635 18471 raft_consensus.cc:515] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.688169 18591 raft_consensus.cc:515] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.688585 18471 leader_election.cc:290] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.688788 18591 leader_election.cc:290] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
W20260812 20:03:58.666021 17883 consensus_peers.cc:597] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c71fd265588a4fccae425a41ef98fb05. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.690243 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.690492 18281 raft_consensus.cc:2468] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.690759 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.690937 18277 raft_consensus.cc:3060] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.691033 17883 leader_election.cc:304] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.691370 18591 raft_consensus.cc:2804] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.691502 18591 raft_consensus.cc:493] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.691606 18591 raft_consensus.cc:3060] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.693771 18125 raft_consensus.cc:2468] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.694046 18277 raft_consensus.cc:2468] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.694147 18406 raft_consensus.cc:515] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.694530 17883 leader_election.cc:304] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.694506 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.694689 18279 raft_consensus.cc:3060] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.694993 18471 raft_consensus.cc:2804] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.695050 18406 leader_election.cc:290] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.695124 18471 raft_consensus.cc:697] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.695390 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.695405 18471 consensus_queue.cc:237] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.695739 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":219,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.652839 18501 ts_tablet_manager.cc:1916] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:03:58.697253 17884 leader_election.cc:343] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.697525 18279 raft_consensus.cc:2468] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.697788 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.698844 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.699364 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:03:58.699594 18207 leader_election.cc:343] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:58.699975 17884 leader_election.cc:343] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.700162 18208 leader_election.cc:304] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
W20260812 20:03:58.700367 17884 leader_election.cc:343] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.700615 18567 raft_consensus.cc:2804] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.700804 18567 raft_consensus.cc:697] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:58.701095 18567 consensus_queue.cc:237] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.701880 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.702466 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6397453a07df4814be9827bb15738081" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
W20260812 20:03:58.702533 18207 leader_election.cc:343] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.702636 17962 raft_consensus.cc:3060] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:03:58.703027 17884 leader_election.cc:343] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.705806 17883 leader_election.cc:304] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.706193 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
W20260812 20:03:58.707067 17884 leader_election.cc:343] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.711334 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.711741 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6397453a07df4814be9827bb15738081" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.711973 18277 raft_consensus.cc:3060] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.713690 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.027s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":27224}
I20260812 20:03:58.714342 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"mutex_wait_us":20621,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.714726 18591 raft_consensus.cc:515] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.715044 18515 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.714915 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.069s user 0.002s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":205,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.715204 18515 ts_tablet_manager.cc:1403] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.123s user 0.015s sys 0.011s
I20260812 20:03:58.715540 18591 leader_election.cc:290] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.715777 18591 raft_consensus.cc:2804] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.715909 18591 raft_consensus.cc:697] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.716174 18515 raft_consensus.cc:359] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.716228 18591 consensus_queue.cc:237] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.716395 18515 raft_consensus.cc:385] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.716490 18515 raft_consensus.cc:740] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.716761 18515 consensus_queue.cc:260] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.716883 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.051s user 0.007s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":724,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1124,"lbm_read_time_us":198,"lbm_reads_lt_1ms":8,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"mutex_wait_us":32807,"reinsert_count":0,"update_count":4}
I20260812 20:03:58.717363 18515 ts_tablet_manager.cc:1434] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260812 20:03:58.717597 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.717952 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.718221 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.718375 18515 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.719305 17962 raft_consensus.cc:2468] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.720448 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.720705 18279 raft_consensus.cc:3060] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.720850 18515 tablet_bootstrap.cc:654] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.721803 18043 leader_election.cc:304] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.722864 18406 raft_consensus.cc:2804] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.723143 18406 raft_consensus.cc:697] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:03:58.723521 18406 consensus_queue.cc:237] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.724228 18279 raft_consensus.cc:2468] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.724761 17883 leader_election.cc:304] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.725481 18379 raft_consensus.cc:2804] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.725682 18379 raft_consensus.cc:697] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.726042 18379 consensus_queue.cc:237] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.726540 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":743,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":65,"lbm_read_time_us":225,"lbm_reads_lt_1ms":12,"lbm_write_time_us":191,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":5}
I20260812 20:03:58.727471 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.731698 18503 ts_tablet_manager.cc:1929] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.727479 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.727706 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.735240 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":46703}
I20260812 20:03:58.735941 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.048s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":166,"lbm_writes_lt_1ms":4,"mutex_wait_us":39178,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.736986 18277 raft_consensus.cc:2468] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:58.737821 18503 log.cc:1199] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/f61d5f81507649418fcd65eee57dc3f3
I20260812 20:03:58.738417 18503 ts_tablet_manager.cc:1950] T f61d5f81507649418fcd65eee57dc3f3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.739585 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44302 (request call id 30) took 1133 ms. Trace:
I20260812 20:03:58.739727 18503 rpcz_store.cc:276] 0812 20:03:57.605801 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.605887 (+ 86us) service_pool.cc:224] Handling call
0812 20:03:58.640194 (+1034307us) ts_tablet_manager.cc:1151] Deleting tablet f61d5f81507649418fcd65eee57dc3f3
0812 20:03:58.640205 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.716342 (+ 76137us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.731579 (+ 15237us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.739518 (+ 7939us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1033996}
I20260812 20:03:58.740211 18503 tablet_replica.cc:333] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.740484 18503 raft_consensus.cc:2243] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.740762 18503 raft_consensus.cc:2272] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.732769 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e7e3cf6c3324a42aa08da0128dda7ca" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.741997 17768 catalog_manager.cc:5719] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.742542 18508 tablet_bootstrap.cc:492] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.742739 18508 ts_tablet_manager.cc:1403] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.022s
W20260812 20:03:58.736671 17884 leader_election.cc:343] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:58.743551 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet f61d5f81507649418fcd65eee57dc3f3 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.744271 18207 leader_election.cc:304] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:58.744580 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.744719 18501 ts_tablet_manager.cc:1929] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.744897 18501 log.cc:1199] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/ba44ce3aac624af697574a4274f38bf5
I20260812 20:03:58.744901 18443 raft_consensus.cc:2749] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:58.745223 18508 raft_consensus.cc:359] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.745551 18508 raft_consensus.cc:385] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.745721 18508 raft_consensus.cc:740] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.746044 18508 consensus_queue.cc:260] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.746172 18503 ts_tablet_manager.cc:1916] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.746905 18508 ts_tablet_manager.cc:1434] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 20:03:58.747308 18508 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:58.748103 18501 ts_tablet_manager.cc:1950] T ba44ce3aac624af697574a4274f38bf5 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.749125 18501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 31) took 1140 ms. Trace:
I20260812 20:03:58.751056 18443 raft_consensus.cc:493] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.751267 18443 raft_consensus.cc:515] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:03:58.751978 17884 consensus_peers.cc:597] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:58.751983 18443 leader_election.cc:290] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.747565 18515 tablet_bootstrap.cc:492] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.753777 18515 ts_tablet_manager.cc:1403] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.002s
I20260812 20:03:58.753854 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f052b8cb7864b5693be4845f8c473ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.754196 18125 raft_consensus.cc:2468] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.754828 18208 leader_election.cc:304] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.754900 18515 raft_consensus.cc:359] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.755108 18515 raft_consensus.cc:385] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.755198 18515 raft_consensus.cc:740] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.755326 18443 raft_consensus.cc:2804] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.755420 18443 raft_consensus.cc:493] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.755529 18443 raft_consensus.cc:3060] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.755511 18515 consensus_queue.cc:260] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.756080 18515 ts_tablet_manager.cc:1434] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:58.757153 18515 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.757911 17767 catalog_manager.cc:5719] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.758780 18443 raft_consensus.cc:515] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.759148 18501 rpcz_store.cc:276] 0812 20:03:57.608819 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.608875 (+ 56us) service_pool.cc:224] Handling call
0812 20:03:58.650438 (+1041563us) ts_tablet_manager.cc:1151] Deleting tablet ba44ce3aac624af697574a4274f38bf5
0812 20:03:58.650457 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.699635 (+ 49178us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.744596 (+ 44961us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.749079 (+ 4483us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041337}
I20260812 20:03:58.759855 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet ba44ce3aac624af697574a4274f38bf5 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.759963 18443 leader_election.cc:290] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:58.760357 18501 tablet_replica.cc:333] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:58.760638 18501 raft_consensus.cc:2243] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.760707 18503 ts_tablet_manager.cc:1929] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.736308 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":16451}
I20260812 20:03:58.760933 18501 raft_consensus.cc:2272] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.760977 18503 log.cc:1199] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/07fbf74c9ba7470ea946e2b437b3d71e
I20260812 20:03:58.761672 18503 ts_tablet_manager.cc:1950] T 07fbf74c9ba7470ea946e2b437b3d71e P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.761189 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f052b8cb7864b5693be4845f8c473ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.762019 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f052b8cb7864b5693be4845f8c473ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.762075 18125 raft_consensus.cc:3060] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.762147 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f052b8cb7864b5693be4845f8c473ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:58.762264 17964 raft_consensus.cc:2468] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:58.764627 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44302 (request call id 29) took 1161 ms. Trace:
I20260812 20:03:58.764842 18503 rpcz_store.cc:276] 0812 20:03:57.603428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.606693 (+ 3265us) service_pool.cc:224] Handling call
0812 20:03:58.740094 (+1133401us) ts_tablet_manager.cc:1151] Deleting tablet 07fbf74c9ba7470ea946e2b437b3d71e
0812 20:03:58.740109 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.757163 (+ 17054us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.760589 (+ 3426us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.764586 (+ 3997us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133085}
I20260812 20:03:58.765195 18125 raft_consensus.cc:2468] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:03:58.765587 18406 consensus_queue.cc:1048] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.765722 18208 leader_election.cc:304] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:58.765421 18503 tablet_replica.cc:333] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.766033 18443 raft_consensus.cc:2804] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.766268 18443 raft_consensus.cc:697] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:03:58.766475 18503 raft_consensus.cc:2243] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:58.766651 18443 consensus_queue.cc:237] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.766830 18503 raft_consensus.cc:2272] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.768064 18503 ts_tablet_manager.cc:1916] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.768556 18501 ts_tablet_manager.cc:1916] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.770740 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 07fbf74c9ba7470ea946e2b437b3d71e (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.737241 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.060s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":16119,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":5929,"lbm_read_time_us":171,"lbm_reads_lt_1ms":8,"lbm_write_time_us":206,"lbm_writes_lt_1ms":4,"mutex_wait_us":18650,"reinsert_count":0,"update_count":3}
I20260812 20:03:58.773248 18508 tablet_bootstrap.cc:654] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.761852 18515 tablet_bootstrap.cc:654] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.774909 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.008s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":724,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":63,"lbm_read_time_us":645,"lbm_reads_lt_1ms":12,"lbm_write_time_us":210,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":4}
I20260812 20:03:58.776463 18503 ts_tablet_manager.cc:1929] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.776693 18503 log.cc:1199] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/ba44ce3aac624af697574a4274f38bf5
I20260812 20:03:58.777340 18503 ts_tablet_manager.cc:1950] T ba44ce3aac624af697574a4274f38bf5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.778649 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44302 (request call id 31) took 1170 ms. Trace:
I20260812 20:03:58.778872 18503 rpcz_store.cc:276] 0812 20:03:57.608549 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.608617 (+ 68us) service_pool.cc:224] Handling call
0812 20:03:58.765273 (+1156656us) ts_tablet_manager.cc:1151] Deleting tablet ba44ce3aac624af697574a4274f38bf5
0812 20:03:58.765285 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.771944 (+ 6659us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.776359 (+ 4415us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.778557 (+ 2198us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1156340}
I20260812 20:03:58.779646 18503 tablet_replica.cc:333] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:58.780165 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet ba44ce3aac624af697574a4274f38bf5 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.780284 18503 raft_consensus.cc:2243] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:58.780865 18503 raft_consensus.cc:2272] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:58.781991 18503 ts_tablet_manager.cc:1916] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:58.786744 18515 tablet_bootstrap.cc:492] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.786739 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.786944 18515 ts_tablet_manager.cc:1403] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.002s
I20260812 20:03:58.787109 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.787520 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.787846 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.787895 18515 raft_consensus.cc:359] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.788126 18515 raft_consensus.cc:385] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.788229 18515 raft_consensus.cc:740] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.788537 18515 consensus_queue.cc:260] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.789234 18515 ts_tablet_manager.cc:1434] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:58.789565 18515 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.790607 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.790975 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.791278 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.791555 18515 tablet_bootstrap.cc:654] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.792351 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.792757 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.793094 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.800866 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.801282 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.801580 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.801908 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.802796 18377 consensus_queue.cc:1048] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.804452 18501 ts_tablet_manager.cc:1929] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.804723 18501 log.cc:1199] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/c71fd265588a4fccae425a41ef98fb05
I20260812 20:03:58.805351 18501 ts_tablet_manager.cc:1950] T c71fd265588a4fccae425a41ef98fb05 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:58.806576 18501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 32) took 1195 ms. Trace:
I20260812 20:03:58.806797 18501 rpcz_store.cc:276] 0812 20:03:57.611457 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.611515 (+ 58us) service_pool.cc:224] Handling call
0812 20:03:58.760267 (+1148752us) ts_tablet_manager.cc:1151] Deleting tablet c71fd265588a4fccae425a41ef98fb05
0812 20:03:58.760281 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.772240 (+ 11959us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.804327 (+ 32087us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.806503 (+ 2176us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148516}
I20260812 20:03:58.807839 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet c71fd265588a4fccae425a41ef98fb05 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.811053 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":17464}
I20260812 20:03:58.811631 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":16164}
I20260812 20:03:58.812212 18379 raft_consensus.cc:493] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.812517 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.107535
I20260812 20:03:58.812469 18379 raft_consensus.cc:515] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.812891 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.107535
I20260812 20:03:58.813460 18379 leader_election.cc:290] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.813371 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.002s sys 0.002s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":200,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.814570 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.815244 18515 tablet_bootstrap.cc:492] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.815462 18515 ts_tablet_manager.cc:1403] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20260812 20:03:58.816141 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.816434 18515 raft_consensus.cc:359] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.816692 18515 raft_consensus.cc:385] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.816811 18515 raft_consensus.cc:740] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.817126 18515 consensus_queue.cc:260] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.816534 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.817925 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.817857 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.818361 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.818516 18515 ts_tablet_manager.cc:1434] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 20:03:58.818467 18281 raft_consensus.cc:2468] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.818838 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:03:58.818887 18515 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.819103 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.819979 17883 leader_election.cc:304] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.820708 18379 raft_consensus.cc:2804] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:58.820860 18515 tablet_bootstrap.cc:654] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.820850 18379 raft_consensus.cc:493] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:58.815830 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.822348 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.823589 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.006s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":211,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.824512 18379 raft_consensus.cc:3060] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.826359 18377 consensus_queue.cc:1048] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.827968 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:58.828074 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":244,"lbm_writes_lt_1ms":4,"mutex_wait_us":7506,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.828243 18125 raft_consensus.cc:2468] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:58.829756 18379 raft_consensus.cc:515] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.832902 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:58.833256 18281 raft_consensus.cc:3060] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.834046 18379 leader_election.cc:290] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.836176 17767 catalog_manager.cc:5719] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.836810 17767 catalog_manager.cc:5719] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.837574 18281 raft_consensus.cc:2468] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.841557 17883 leader_election.cc:304] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:58.842298 18591 raft_consensus.cc:2804] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:58.842406 18377 consensus_queue.cc:1048] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.842605 18591 raft_consensus.cc:697] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:58.842979 18591 consensus_queue.cc:237] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.849932 18377 consensus_queue.cc:1048] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.850876 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.851258 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfafbefa0be43d694d032f733f1d7b1" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:58.851500 18125 raft_consensus.cc:3060] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.851823 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.854836 18125 raft_consensus.cc:2468] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:58.859982 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.860414 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.861485 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.866925 18503 ts_tablet_manager.cc:1929] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:58.867170 18503 log.cc:1199] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/c71fd265588a4fccae425a41ef98fb05
I20260812 20:03:58.867863 18503 ts_tablet_manager.cc:1950] T c71fd265588a4fccae425a41ef98fb05 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:58.872186 18503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44302 (request call id 32) took 1261 ms. Trace:
I20260812 20:03:58.872462 18503 rpcz_store.cc:276] 0812 20:03:57.610732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:03:57.618608 (+ 7876us) service_pool.cc:224] Handling call
0812 20:03:58.779510 (+1160902us) ts_tablet_manager.cc:1151] Deleting tablet c71fd265588a4fccae425a41ef98fb05
0812 20:03:58.779526 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:03:58.848701 (+ 69175us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.866787 (+ 18086us) tablet_metadata.cc:665] Metadata flushed
0812 20:03:58.872110 (+ 5323us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160581}
I20260812 20:03:58.879627 17767 catalog_manager.cc:5719] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:58.880775 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.883103 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet c71fd265588a4fccae425a41ef98fb05 (table table-0 [id=fdf0b43d520b4768a902707b8b2719bc]) successfully deleted
I20260812 20:03:58.888938 18515 tablet_bootstrap.cc:492] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.889106 18515 ts_tablet_manager.cc:1403] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.014s
I20260812 20:03:58.890276 18515 raft_consensus.cc:359] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.890532 18515 raft_consensus.cc:385] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.890655 18515 raft_consensus.cc:740] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.890956 18515 consensus_queue.cc:260] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.892429 18515 ts_tablet_manager.cc:1434] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:03:58.892746 18515 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:58.895977 18379 consensus_queue.cc:1048] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.896948 18515 tablet_bootstrap.cc:654] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:58.880723 18508 tablet_bootstrap.cc:492] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:58.902295 18508 ts_tablet_manager.cc:1403] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.155s user 0.021s sys 0.024s
I20260812 20:03:58.903156 18508 raft_consensus.cc:359] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.903409 18508 raft_consensus.cc:385] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.903496 18508 raft_consensus.cc:740] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.903789 18508 consensus_queue.cc:260] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.903901 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.091s user 0.010s sys 0.006s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":12661,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":6986,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":1493,"lbm_read_time_us":957,"lbm_reads_lt_1ms":33,"lbm_write_time_us":479,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":7}
I20260812 20:03:58.909425 18508 ts_tablet_manager.cc:1434] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260812 20:03:58.912369 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.050s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":50207}
I20260812 20:03:58.912689 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.051s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":50495}
I20260812 20:03:58.915030 17767 catalog_manager.cc:5719] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:58.917770 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.005s sys 0.000s Metrics: {"bytes_written":481,"delete_count":0,"lbm_write_time_us":254,"lbm_writes_lt_1ms":4,"mutex_wait_us":41957,"reinsert_count":0,"update_count":3}
I20260812 20:03:58.918967 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.055s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":54586}
I20260812 20:03:58.919471 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":33593}
I20260812 20:03:58.919853 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.068s user 0.003s sys 0.003s Metrics: {"bytes_written":481,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"mutex_wait_us":53461,"reinsert_count":0,"update_count":3}
I20260812 20:03:58.920277 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.102s user 0.008s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":773,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":16794,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1271,"lbm_read_time_us":157,"lbm_reads_lt_1ms":8,"lbm_write_time_us":180,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":6}
I20260812 20:03:58.920832 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.921082 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.921341 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.924579 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.924823 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.921581 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.925211 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.926004 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.926288 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.926555 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.928696 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.929028 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.929246 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.929461 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.931432 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.931758 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.931969 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.931928 17768 catalog_manager.cc:5719] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:03:58.932384 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.932829 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.933106 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.941663 18515 tablet_bootstrap.cc:492] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:58.941501 18591 consensus_queue.cc:1048] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.941872 18515 ts_tablet_manager.cc:1403] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.006s
I20260812 20:03:58.942871 18515 raft_consensus.cc:359] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.943113 18515 raft_consensus.cc:385] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:58.943216 18515 raft_consensus.cc:740] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:58.943532 18515 consensus_queue.cc:260] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.944258 18515 ts_tablet_manager.cc:1434] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:58.950582 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.006s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":715,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":53,"lbm_read_time_us":153,"lbm_reads_lt_1ms":8,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"mutex_wait_us":7072,"reinsert_count":0,"update_count":4}
I20260812 20:03:58.951316 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:58.951817 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.004s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.952206 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.952450 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.953513 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.953832 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.954034 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.956058 18567 consensus_queue.cc:1048] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.958068 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.036s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":851,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":131,"lbm_reads_lt_1ms":8,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":9}
I20260812 20:03:58.959077 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.182447
I20260812 20:03:58.962855 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.963478 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.964886 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.006s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"mutex_wait_us":31825,"reinsert_count":0,"update_count":2}
I20260812 20:03:58.965720 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.965246 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":10180}
I20260812 20:03:58.966157 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.965835 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.966533 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:58.967021 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":192,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:03:58.969123 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:58.972937 18570 consensus_queue.cc:1048] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:58.974851 18377 raft_consensus.cc:493] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:58.975034 18377 raft_consensus.cc:515] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:58.975744 18377 leader_election.cc:290] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:58.975414 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":735,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":53,"lbm_read_time_us":131,"lbm_reads_lt_1ms":8,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"mutex_wait_us":12123,"reinsert_count":0,"update_count":5}
I20260812 20:03:58.976298 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:58.977052 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e1ec98f1d914666bc5e02306ee16d13" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:58.977357 18281 raft_consensus.cc:2393] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d222b63b24184ca6bc530a1547fe3297 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:03:58.977954 18043 leader_election.cc:304] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:58.978626 18377 raft_consensus.cc:3060] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:58.981860 18377 raft_consensus.cc:2749] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:58.985846 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.004s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":904,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":228,"lbm_read_time_us":412,"lbm_reads_lt_1ms":8,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":11}
I20260812 20:03:58.993288 18379 consensus_queue.cc:1048] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.001943 18567 raft_consensus.cc:493] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.002158 18567 raft_consensus.cc:515] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.003332 18567 leader_election.cc:290] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.005439 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.007879 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2eb8accc87ef4ee4a5186719c042db42" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.008870 18126 raft_consensus.cc:2393] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:03:59.009409 18208 leader_election.cc:304] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.010078 18567 raft_consensus.cc:3060] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.013211 18567 raft_consensus.cc:2749] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.049371 18591 consensus_queue.cc:1048] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.058148 18379 consensus_queue.cc:1048] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.072271 18443 consensus_queue.cc:1048] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.086036 18443 consensus_queue.cc:1048] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.090098 18125 raft_consensus.cc:3060] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.093483 18377 raft_consensus.cc:493] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.093744 18377 raft_consensus.cc:515] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.094259 18591 consensus_queue.cc:1048] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.094705 18377 leader_election.cc:290] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.095394 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:59.095618 18281 raft_consensus.cc:2468] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:03:59.096192 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.097126 18043 leader_election.cc:304] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.098660 18377 raft_consensus.cc:2804] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 20:03:59.098948 18377 raft_consensus.cc:493] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f20226c1597d481a84f0c2fe53f0bb65)
I20260812 20:03:59.099114 18377 raft_consensus.cc:3060] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Advancing to term 2
I20260812 20:03:59.103008 18377 raft_consensus.cc:515] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.105212 18379 consensus_queue.cc:1048] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.106490 18377 leader_election.cc:290] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 2 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.107302 18125 raft_consensus.cc:1240] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: Rejecting Update request from peer f20226c1597d481a84f0c2fe53f0bb65 for earlier term 1. Current term is 2. Ops: []
I20260812 20:03:59.107441 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:59.108453 18591 consensus_queue.cc:1059] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.108870 18379 raft_consensus.cc:3055] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 20:03:59.108984 18379 raft_consensus.cc:740] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:59.109295 18379 consensus_queue.cc:260] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.109542 18379 raft_consensus.cc:3060] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Advancing to term 2
I20260812 20:03:59.109519 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5000368644a64059af5235482fe2e79c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:59.114424 18043 leader_election.cc:304] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.115607 18377 raft_consensus.cc:2749] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 20:03:59.139008 17769 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-1" } modify_external_catalogs: true
I20260812 20:03:59.139281 17769 catalog_manager.cc:2781] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-1" } modify_external_catalogs: true
I20260812 20:03:59.149645 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.150058 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.150548 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.150575 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.150956 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.151543 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.151985 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.152516 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.154196 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet a2950ec0320c432887522c75de50d8c5
I20260812 20:03:59.155325 18251 tablet_service.cc:1558] Processing DeleteTablet for tablet a2950ec0320c432887522c75de50d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.155437 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 3e5c8e0a860641168abfaee4c7216fd7
I20260812 20:03:59.155411 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet a2950ec0320c432887522c75de50d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.155525 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet a2950ec0320c432887522c75de50d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.155807 18503 tablet_replica.cc:333] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.155798 18501 tablet_replica.cc:333] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.156065 18503 raft_consensus.cc:2243] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.156093 18501 raft_consensus.cc:2243] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.156292 18503 raft_consensus.cc:2272] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.156615 18501 raft_consensus.cc:2272] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.157454 18503 ts_tablet_manager.cc:1916] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.157783 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.001s Metrics: {"bytes_written":742,"delete_count":0,"lbm_write_time_us":212,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":8}
I20260812 20:03:59.158018 18501 ts_tablet_manager.cc:1916] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.158444 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.159643 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.160043 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.160391 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.160868 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.640429
I20260812 20:03:59.161178 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.640429
I20260812 20:03:59.161769 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.165417 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":13147}
I20260812 20:03:59.167284 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":12994}
I20260812 20:03:59.167934 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.142323
I20260812 20:03:59.168293 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.142323
I20260812 20:03:59.168309 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.169703 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":14134}
I20260812 20:03:59.170069 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.170424 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.170897 18501 ts_tablet_manager.cc:1929] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.171068 18501 log.cc:1199] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/a2950ec0320c432887522c75de50d8c5
I20260812 20:03:59.171500 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:03:59.172521 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet bb77a5184b8a459db4499e77cd2df733
I20260812 20:03:59.172513 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e5c8e0a860641168abfaee4c7216fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.173534 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet bb77a5184b8a459db4499e77cd2df733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.174386 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet bb77a5184b8a459db4499e77cd2df733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.175544 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet aad9b9e5ec944a1aa309cc9332b2dbca
I20260812 20:03:59.175561 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet bb77a5184b8a459db4499e77cd2df733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.176553 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.176980 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.177330 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.177539 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.002s sys 0.003s Metrics: {"bytes_written":950,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:03:59.178539 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.001s sys 0.004s Metrics: {"bytes_written":631,"delete_count":0,"lbm_write_time_us":178,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":9}
I20260812 20:03:59.178629 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet aad9b9e5ec944a1aa309cc9332b2dbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.178956 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 489ea23ba78a47bda493198355f9b340
I20260812 20:03:59.180055 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 489ea23ba78a47bda493198355f9b340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.180612 18501 ts_tablet_manager.cc:1950] T a2950ec0320c432887522c75de50d8c5 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.182361 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet a2950ec0320c432887522c75de50d8c5 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.182863 18591 consensus_queue.cc:1048] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.183681 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 489ea23ba78a47bda493198355f9b340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.184834 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 489ea23ba78a47bda493198355f9b340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.185127 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet c5c9d056274f4c1ab24c5f317981d016
I20260812 20:03:59.185202 18503 ts_tablet_manager.cc:1929] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.185372 18503 log.cc:1199] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/a2950ec0320c432887522c75de50d8c5
I20260812 20:03:59.185999 18503 ts_tablet_manager.cc:1950] T a2950ec0320c432887522c75de50d8c5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.186273 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet c5c9d056274f4c1ab24c5f317981d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.186901 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet c5c9d056274f4c1ab24c5f317981d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.186988 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":13413,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":102,"lbm_read_time_us":1201,"lbm_reads_lt_1ms":33,"lbm_write_time_us":355,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":22}
I20260812 20:03:59.188052 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b607a7c7ca104dbca81ab5ddae140a58
I20260812 20:03:59.188100 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet c5c9d056274f4c1ab24c5f317981d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.188902 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.189003 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.189801 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet b607a7c7ca104dbca81ab5ddae140a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.189853 18501 tablet_replica.cc:333] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.190196 18501 raft_consensus.cc:2243] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.190429 18503 tablet_replica.cc:333] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.190536 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet a2950ec0320c432887522c75de50d8c5 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.190640 18503 raft_consensus.cc:2243] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.190840 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 2eb8accc87ef4ee4a5186719c042db42
I20260812 20:03:59.190879 18503 raft_consensus.cc:2272] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.191749 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.192186 18503 ts_tablet_manager.cc:1916] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.192624 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.192975 18377 consensus_queue.cc:1048] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.193742 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 2eb8accc87ef4ee4a5186719c042db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.194099 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 2e1ec98f1d914666bc5e02306ee16d13
I20260812 20:03:59.194551 18501 raft_consensus.cc:2272] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.195040 18633 tablet_replica.cc:333] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.195319 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.012s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":13025,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":293,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":633,"lbm_read_time_us":568,"lbm_reads_lt_1ms":27,"lbm_write_time_us":517,"lbm_writes_lt_1ms":9,"mutex_wait_us":4052,"peak_mem_usage":32768,"reinsert_count":0,"update_count":28}
I20260812 20:03:59.196766 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.197465 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.198149 18633 raft_consensus.cc:2243] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.198794 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5000368644a64059af5235482fe2e79c
I20260812 20:03:59.200103 18503 ts_tablet_manager.cc:1929] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.200265 18503 log.cc:1199] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/3e5c8e0a860641168abfaee4c7216fd7
I20260812 20:03:59.200829 18503 ts_tablet_manager.cc:1950] T 3e5c8e0a860641168abfaee4c7216fd7 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.202023 18421 consensus_queue.cc:1048] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.202700 18503 tablet_replica.cc:333] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.202948 18503 raft_consensus.cc:2243] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.203332 18503 raft_consensus.cc:2272] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.199802 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e1ec98f1d914666bc5e02306ee16d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.205027 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.205127 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.205547 18633 raft_consensus.cc:2272] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.202219 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 3e5c8e0a860641168abfaee4c7216fd7 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.207162 18501 ts_tablet_manager.cc:1916] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.207432 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 5000368644a64059af5235482fe2e79c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.209214 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 5000368644a64059af5235482fe2e79c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.210039 18251 tablet_service.cc:1558] Processing DeleteTablet for tablet 5000368644a64059af5235482fe2e79c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.210805 18503 ts_tablet_manager.cc:1916] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.212607 18633 ts_tablet_manager.cc:1916] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.216085 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 7e7e3cf6c3324a42aa08da0128dda7ca
I20260812 20:03:59.217661 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.218295 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.219799 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 7e7e3cf6c3324a42aa08da0128dda7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.220112 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 9f052b8cb7864b5693be4845f8c473ac
I20260812 20:03:59.220738 18481 consensus_queue.cc:1048] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.221526 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 9f052b8cb7864b5693be4845f8c473ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.221702 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 9f052b8cb7864b5693be4845f8c473ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.221963 18251 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.222419 18421 raft_consensus.cc:493] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.222620 18421 raft_consensus.cc:515] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.222848 18503 ts_tablet_manager.cc:1929] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.223014 18503 log.cc:1199] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/bb77a5184b8a459db4499e77cd2df733
I20260812 20:03:59.223428 18421 leader_election.cc:290] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.223537 18503 ts_tablet_manager.cc:1950] T bb77a5184b8a459db4499e77cd2df733 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.223819 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.224038 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:59.224629 18503 tablet_replica.cc:333] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.224845 18503 raft_consensus.cc:2243] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.225049 18503 raft_consensus.cc:2272] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.225399 18633 ts_tablet_manager.cc:1929] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.225462 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet bb77a5184b8a459db4499e77cd2df733 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.226585 18503 ts_tablet_manager.cc:1916] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.226743 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b607a7c7ca104dbca81ab5ddae140a58" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.227007 18125 raft_consensus.cc:2393] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:03:59.227777 17884 leader_election.cc:304] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:59.228284 18421 raft_consensus.cc:3060] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.229861 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 6397453a07df4814be9827bb15738081
I20260812 20:03:59.231006 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.231711 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:03:59.232187 18443 consensus_queue.cc:1048] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.232336 18633 log.cc:1199] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/a2950ec0320c432887522c75de50d8c5
I20260812 20:03:59.232671 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 8dfafbefa0be43d694d032f733f1d7b1
I20260812 20:03:59.232658 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.232909 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.233371 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.233767 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 44959dd27d9642f0ad8ddf8395887083
I20260812 20:03:59.233739 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.233752 18633 ts_tablet_manager.cc:1950] T a2950ec0320c432887522c75de50d8c5 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.234114 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.234891 18094 tablet_service.cc:1558] Processing DeleteTablet for tablet 44959dd27d9642f0ad8ddf8395887083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:03:59.235044 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 6397453a07df4814be9827bb15738081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.236316 18240 tablet_service.cc:1558] Processing DeleteTablet for tablet 44959dd27d9642f0ad8ddf8395887083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.236610 18248 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.229722 18251 tablet_service.cc:1558] Processing DeleteTablet for tablet 9f052b8cb7864b5693be4845f8c473ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:03:59.237370 18633 tablet_replica.cc:333] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.237699 18633 raft_consensus.cc:2243] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.237943 18633 raft_consensus.cc:2272] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.239070 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet a2950ec0320c432887522c75de50d8c5 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.239971 18633 ts_tablet_manager.cc:1916] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.240742 17929 tablet_service.cc:1558] Processing DeleteTablet for tablet 44959dd27d9642f0ad8ddf8395887083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.240955 18421 raft_consensus.cc:2749] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.241110 17931 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dfafbefa0be43d694d032f733f1d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.242100 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 77032e8e8636495a9dfdfa1ce5d85021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:03:59 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:03:59.240394 17769 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38702:
name: "table-2"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260812 20:03:59.244355 17961 raft_consensus.cc:3060] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.247892 18503 ts_tablet_manager.cc:1929] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.248041 18503 log.cc:1199] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/aad9b9e5ec944a1aa309cc9332b2dbca
I20260812 20:03:59.248593 18503 ts_tablet_manager.cc:1950] T aad9b9e5ec944a1aa309cc9332b2dbca P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.249432 18443 consensus_queue.cc:1048] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.250254 18503 tablet_replica.cc:333] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.250497 18503 raft_consensus.cc:2243] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.250588 18633 ts_tablet_manager.cc:1929] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.250679 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet aad9b9e5ec944a1aa309cc9332b2dbca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.250748 18633 log.cc:1199] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/3e5c8e0a860641168abfaee4c7216fd7
I20260812 20:03:59.250743 18503 raft_consensus.cc:2272] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.251224 18633 ts_tablet_manager.cc:1950] T 3e5c8e0a860641168abfaee4c7216fd7 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.252465 18503 ts_tablet_manager.cc:1916] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.253008 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 3e5c8e0a860641168abfaee4c7216fd7 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.253711 18633 tablet_replica.cc:333] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.253985 18633 raft_consensus.cc:2243] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.254264 18633 raft_consensus.cc:2272] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.255378 18633 ts_tablet_manager.cc:1916] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:03:59.255656 17769 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 20:03:59.263013 18633 ts_tablet_manager.cc:1929] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.263322 18633 log.cc:1199] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/bb77a5184b8a459db4499e77cd2df733
I20260812 20:03:59.263976 18633 ts_tablet_manager.cc:1950] T bb77a5184b8a459db4499e77cd2df733 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.265365 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet bb77a5184b8a459db4499e77cd2df733 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.265846 18633 tablet_replica.cc:333] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.266137 18633 raft_consensus.cc:2243] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.266369 18633 raft_consensus.cc:2272] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.266299 18421 consensus_queue.cc:1048] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.267866 18633 ts_tablet_manager.cc:1916] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.268792 18503 ts_tablet_manager.cc:1929] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.269019 18503 log.cc:1199] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/489ea23ba78a47bda493198355f9b340
I20260812 20:03:59.269160 18501 ts_tablet_manager.cc:1929] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.269374 18501 log.cc:1199] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/3e5c8e0a860641168abfaee4c7216fd7
I20260812 20:03:59.269712 18503 ts_tablet_manager.cc:1950] T 489ea23ba78a47bda493198355f9b340 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.269878 18501 ts_tablet_manager.cc:1950] T 3e5c8e0a860641168abfaee4c7216fd7 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.271265 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 3e5c8e0a860641168abfaee4c7216fd7 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.271309 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 489ea23ba78a47bda493198355f9b340 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.271445 18503 tablet_replica.cc:333] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.271667 18503 raft_consensus.cc:2243] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.272104 18503 raft_consensus.cc:2272] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.273902 18503 ts_tablet_manager.cc:1916] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.277925 18501 tablet_replica.cc:333] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.278177 18501 raft_consensus.cc:2243] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.278417 18501 raft_consensus.cc:2272] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.279745 18633 ts_tablet_manager.cc:1929] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.279996 18633 log.cc:1199] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/aad9b9e5ec944a1aa309cc9332b2dbca
I20260812 20:03:59.280169 18501 ts_tablet_manager.cc:1916] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.280586 18633 ts_tablet_manager.cc:1950] T aad9b9e5ec944a1aa309cc9332b2dbca P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.282014 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet aad9b9e5ec944a1aa309cc9332b2dbca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.283529 18633 tablet_replica.cc:333] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.283597 18591 consensus_queue.cc:1048] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.283866 18503 ts_tablet_manager.cc:1929] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.283917 18633 raft_consensus.cc:2243] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.284019 18503 log.cc:1199] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/c5c9d056274f4c1ab24c5f317981d016
I20260812 20:03:59.284504 18633 raft_consensus.cc:2272] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.284591 18503 ts_tablet_manager.cc:1950] T c5c9d056274f4c1ab24c5f317981d016 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.285657 18503 tablet_replica.cc:333] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.285769 18633 ts_tablet_manager.cc:1916] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.285908 18503 raft_consensus.cc:2243] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.286087 18503 raft_consensus.cc:2272] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.287276 18503 ts_tablet_manager.cc:1916] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.290021 18501 ts_tablet_manager.cc:1929] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.290539 18591 consensus_queue.cc:1048] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.292181 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.292500 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.288775 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet c5c9d056274f4c1ab24c5f317981d016 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.301483 18591 consensus_queue.cc:1048] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 20:03:59.302106 18207 consensus_peers.cc:597] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:59.308429 18481 raft_consensus.cc:493] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.308678 18481 raft_consensus.cc:515] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.308909 18127 raft_consensus.cc:3060] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.309453 18481 leader_election.cc:290] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.310005 18633 ts_tablet_manager.cc:1929] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.310168 18633 log.cc:1199] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/489ea23ba78a47bda493198355f9b340
I20260812 20:03:59.310667 18633 ts_tablet_manager.cc:1950] T 489ea23ba78a47bda493198355f9b340 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.312172 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 489ea23ba78a47bda493198355f9b340 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.312724 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c9d056274f4c1ab24c5f317981d016" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:59.313143 18633 tablet_replica.cc:333] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.313441 18633 raft_consensus.cc:2243] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.313762 18633 raft_consensus.cc:2272] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.315459 18633 ts_tablet_manager.cc:1916] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.317600 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.318105 18501 log.cc:1199] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/bb77a5184b8a459db4499e77cd2df733
I20260812 20:03:59.318195 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd14ad61b704e56a9dc1f37ce97925c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:03:59.318567 18043 leader_election.cc:343] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Not found: Tablet not found: c5c9d056274f4c1ab24c5f317981d016
I20260812 20:03:59.318764 18043 leader_election.cc:304] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.319140 18481 raft_consensus.cc:2749] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.320959 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.321483 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd14ad61b704e56a9dc1f37ce97925c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.322945 18379 consensus_queue.cc:1048] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.326051 17931 tablet_service.cc:1511] Processing CreateTablet for tablet b4a4e1ea1f52461c8aec59967842b390 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.328655 18515 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.328770 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.329309 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18b5f38beca5449ebd3f8bcb0f0284d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.331097 18515 tablet_bootstrap.cc:654] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.332719 18503 ts_tablet_manager.cc:1929] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:59.332885 18503 log.cc:1199] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b607a7c7ca104dbca81ab5ddae140a58
I20260812 20:03:59.333287 18503 ts_tablet_manager.cc:1950] T b607a7c7ca104dbca81ab5ddae140a58 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.334038 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 06b17a7a289a449397777c720ca2af7b (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.334573 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b17a7a289a449397777c720ca2af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.337143 18501 ts_tablet_manager.cc:1950] T bb77a5184b8a459db4499e77cd2df733 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.338132 18090 tablet_service.cc:1511] Processing CreateTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.338346 18501 tablet_replica.cc:333] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.338619 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54ad4368314d4cb8713bfb2ef2e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.339838 18515 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.340010 18515 ts_tablet_manager.cc:1403] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20260812 20:03:59.341261 18515 raft_consensus.cc:359] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.341495 18515 raft_consensus.cc:385] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.341673 18515 raft_consensus.cc:740] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.341943 18097 tablet_service.cc:1511] Processing CreateTablet for tablet b4a4e1ea1f52461c8aec59967842b390 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.342020 18515 consensus_queue.cc:260] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.342456 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a4e1ea1f52461c8aec59967842b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.342739 18515 ts_tablet_manager.cc:1434] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:59.338634 18501 raft_consensus.cc:2243] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.346621 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a4e1ea1f52461c8aec59967842b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.346971 18515 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.348923 18515 tablet_bootstrap.cc:654] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.349905 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.057s user 0.017s sys 0.000s Metrics: {"bytes_written":697,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1138,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1287,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"update_count":16}
I20260812 20:03:59.352097 18089 tablet_service.cc:1511] Processing CreateTablet for tablet cb5c495f458246c08c470376ab0dae29 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.352311 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet bb77a5184b8a459db4499e77cd2df733 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.352610 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5c495f458246c08c470376ab0dae29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.353899 18087 tablet_service.cc:1511] Processing CreateTablet for tablet b2328bed2bc24cf29b316636d7f493ba (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.354360 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2328bed2bc24cf29b316636d7f493ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.355621 18104 tablet_service.cc:1511] Processing CreateTablet for tablet f53efb0b87354de88e17f80c56b9eec5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.356030 18104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f53efb0b87354de88e17f80c56b9eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.355203 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:59.357389 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:59.357697 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:59.357992 18091 tablet_service.cc:1511] Processing CreateTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.358351 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0229dd61d8b419ca2c4ecbf1940d6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.358470 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20260812 20:03:59.358896 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.359210 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.359992 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.304489
I20260812 20:03:59.360567 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.304489
I20260812 20:03:59.361088 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.304489
I20260812 20:03:59.362006 18100 tablet_service.cc:1511] Processing CreateTablet for tablet 5aac4ecab28843558e26507a083afd4e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.362432 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aac4ecab28843558e26507a083afd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.364128 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 8b30713b37a9473395eb907e16e77c7c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.364629 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30713b37a9473395eb907e16e77c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.365931 18508 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:59.367491 18481 raft_consensus.cc:493] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.367664 18481 raft_consensus.cc:515] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.368464 18481 leader_election.cc:290] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.368629 18090 tablet_service.cc:1511] Processing CreateTablet for tablet efa6bab2829f436ea657c5e78a693d16 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.369038 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6bab2829f436ea657c5e78a693d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.372834 18508 tablet_bootstrap.cc:654] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.373165 18094 tablet_service.cc:1511] Processing CreateTablet for tablet c17c36cd2a6648c49342725074cdc96e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.373670 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17c36cd2a6648c49342725074cdc96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.374056 18251 tablet_service.cc:1511] Processing CreateTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.374605 18251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd14ad61b704e56a9dc1f37ce97925c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.376291 17920 tablet_service.cc:1511] Processing CreateTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.376865 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0229dd61d8b419ca2c4ecbf1940d6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.377319 17929 tablet_service.cc:1511] Processing CreateTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.377938 17924 tablet_service.cc:1511] Processing CreateTablet for tablet efa6bab2829f436ea657c5e78a693d16 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.378136 17931 tablet_service.cc:1511] Processing CreateTablet for tablet 06b17a7a289a449397777c720ca2af7b (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.378460 17921 tablet_service.cc:1511] Processing CreateTablet for tablet c17c36cd2a6648c49342725074cdc96e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
W20260812 20:03:59.378552 18043 leader_election.cc:343] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Not found: Tablet not found: 3bd14ad61b704e56a9dc1f37ce97925c
I20260812 20:03:59.379078 17932 tablet_service.cc:1511] Processing CreateTablet for tablet f53efb0b87354de88e17f80c56b9eec5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.379694 17938 tablet_service.cc:1511] Processing CreateTablet for tablet 5aac4ecab28843558e26507a083afd4e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.380358 17927 tablet_service.cc:1511] Processing CreateTablet for tablet b2328bed2bc24cf29b316636d7f493ba (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.380867 17926 tablet_service.cc:1511] Processing CreateTablet for tablet 8b30713b37a9473395eb907e16e77c7c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.381726 18248 tablet_service.cc:1511] Processing CreateTablet for tablet b4a4e1ea1f52461c8aec59967842b390 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.377319 17922 tablet_service.cc:1511] Processing CreateTablet for tablet cb5c495f458246c08c470376ab0dae29 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.382229 18248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a4e1ea1f52461c8aec59967842b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.382457 17929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54ad4368314d4cb8713bfb2ef2e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.386046 18443 consensus_queue.cc:1048] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20260812 20:03:59.387358 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd14ad61b704e56a9dc1f37ce97925c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.388834 18501 raft_consensus.cc:2272] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.390340 18501 ts_tablet_manager.cc:1916] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.393388 18243 tablet_service.cc:1511] Processing CreateTablet for tablet cb5c495f458246c08c470376ab0dae29 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.393900 18243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5c495f458246c08c470376ab0dae29. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:03:59.396165 18043 leader_election.cc:343] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.396333 18043 leader_election.cc:304] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.397708 18244 tablet_service.cc:1511] Processing CreateTablet for tablet efa6bab2829f436ea657c5e78a693d16 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.398072 18406 raft_consensus.cc:2749] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.398188 18244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6bab2829f436ea657c5e78a693d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.399132 18249 tablet_service.cc:1511] Processing CreateTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.399478 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54ad4368314d4cb8713bfb2ef2e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.398932 18240 tablet_service.cc:1511] Processing CreateTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.400115 18257 tablet_service.cc:1511] Processing CreateTablet for tablet f53efb0b87354de88e17f80c56b9eec5 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.401686 18254 tablet_service.cc:1511] Processing CreateTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.402064 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.402452 18241 tablet_service.cc:1511] Processing CreateTablet for tablet 5aac4ecab28843558e26507a083afd4e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.400665 18255 tablet_service.cc:1511] Processing CreateTablet for tablet 06b17a7a289a449397777c720ca2af7b (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.403658 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18b5f38beca5449ebd3f8bcb0f0284d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.406888 17935 tablet_service.cc:1511] Processing CreateTablet for tablet d8750126e2be4861b6a61b34af927f54 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.409713 18242 tablet_service.cc:1511] Processing CreateTablet for tablet c17c36cd2a6648c49342725074cdc96e (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.410104 18255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b17a7a289a449397777c720ca2af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.414572 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b607a7c7ca104dbca81ab5ddae140a58 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.414808 18247 tablet_service.cc:1511] Processing CreateTablet for tablet b2328bed2bc24cf29b316636d7f493ba (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.415438 17935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8750126e2be4861b6a61b34af927f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.415793 18471 consensus_queue.cc:1048] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:03:59.417052 17933 tablet_service.cc:1511] Processing CreateTablet for tablet 006049a466af4197a4f71989bf9e1b50 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.417503 17933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006049a466af4197a4f71989bf9e1b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.419203 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 8b30713b37a9473395eb907e16e77c7c (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.420413 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17c36cd2a6648c49342725074cdc96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.422120 18245 tablet_service.cc:1511] Processing CreateTablet for tablet d8750126e2be4861b6a61b34af927f54 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.422482 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30713b37a9473395eb907e16e77c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.422528 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18b5f38beca5449ebd3f8bcb0f0284d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.422537 18100 tablet_service.cc:1511] Processing CreateTablet for tablet d8750126e2be4861b6a61b34af927f54 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.422948 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8750126e2be4861b6a61b34af927f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.430437 17926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30713b37a9473395eb907e16e77c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.433923 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17c36cd2a6648c49342725074cdc96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.435844 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.143s user 0.016s sys 0.000s Metrics: {"bytes_written":697,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1299,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":563,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2560,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"update_count":16}
I20260812 20:03:59.436966 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:59.437340 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:03:59.437856 17924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6bab2829f436ea657c5e78a693d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.437903 18250 tablet_service.cc:1511] Processing CreateTablet for tablet 006049a466af4197a4f71989bf9e1b50 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.438066 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260812 20:03:59.438478 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.439150 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260812 20:03:59.441936 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5c495f458246c08c470376ab0dae29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.441983 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.301904
I20260812 20:03:59.444962 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1404f2da764dfd8797dfaad98e954f (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.445495 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1404f2da764dfd8797dfaad98e954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.445891 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aac4ecab28843558e26507a083afd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.448185 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2328bed2bc24cf29b316636d7f493ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.448596 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f53efb0b87354de88e17f80c56b9eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.458175 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b17a7a289a449397777c720ca2af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.458583 18245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8750126e2be4861b6a61b34af927f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.459473 18254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0229dd61d8b419ca2c4ecbf1940d6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.460497 18250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006049a466af4197a4f71989bf9e1b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.461131 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f53efb0b87354de88e17f80c56b9eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.461745 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2328bed2bc24cf29b316636d7f493ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.462987 18241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aac4ecab28843558e26507a083afd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.475238 17667 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.477164 18260 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1404f2da764dfd8797dfaad98e954f (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.477226 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.006s sys 0.005s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1677,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":9269,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":929,"lbm_read_time_us":479,"lbm_reads_lt_1ms":21,"lbm_write_time_us":614,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260812 20:03:59.477823 18260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1404f2da764dfd8797dfaad98e954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.487388 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.122s user 0.014s sys 0.008s Metrics: {"dirs.queue_time_us":7465,"dirs.run_cpu_time_us":970,"dirs.run_wall_time_us":7913,"lbm_write_time_us":369,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"thread_start_us":103749,"threads_started":2}
I20260812 20:03:59.494236 18503 tablet_replica.cc:333] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.494577 18503 raft_consensus.cc:2243] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.490365 17668 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.495091 18503 raft_consensus.cc:2272] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.496265 18503 ts_tablet_manager.cc:1916] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.499253 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.130s user 0.008s sys 0.015s Metrics: {"dirs.queue_time_us":5896,"dirs.run_cpu_time_us":935,"dirs.run_wall_time_us":4170,"lbm_write_time_us":448,"lbm_writes_lt_1ms":7,"mutex_wait_us":2559,"num_input_rowsets":0}
I20260812 20:03:59.496817 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1404f2da764dfd8797dfaad98e954f (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.493664 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 006049a466af4197a4f71989bf9e1b50 (DEFAULT_TABLE table=table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:03:59.501652 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006049a466af4197a4f71989bf9e1b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.504328 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1404f2da764dfd8797dfaad98e954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:03:59.512265 18503 ts_tablet_manager.cc:1929] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.517174 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2852 bytes on disk
I20260812 20:03:59.517522 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2852 bytes on disk
I20260812 20:03:59.517813 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2852 bytes on disk
I20260812 20:03:59.518807 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20260812 20:03:59.519301 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.519573 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.521787 18503 log.cc:1199] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/2eb8accc87ef4ee4a5186719c042db42
I20260812 20:03:59.522531 18503 ts_tablet_manager.cc:1950] T 2eb8accc87ef4ee4a5186719c042db42 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.523991 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 2eb8accc87ef4ee4a5186719c042db42 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.524500 18503 tablet_replica.cc:333] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.524830 18503 raft_consensus.cc:2243] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.525276 18503 raft_consensus.cc:2272] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.527889 18503 ts_tablet_manager.cc:1916] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.534991 18503 ts_tablet_manager.cc:1929] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.537509 18501 ts_tablet_manager.cc:1929] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.542076 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.181s user 0.026s sys 0.015s Metrics: {"bytes_written":25078,"cfile_cache_hit":15,"cfile_cache_hit_bytes":15212,"cfile_cache_miss":37,"cfile_cache_miss_bytes":23468,"cfile_init":10,"delta_iterators_relevant":10,"dirs.queue_time_us":37087,"dirs.run_cpu_time_us":1506,"dirs.run_wall_time_us":7499,"drs_written":1,"lbm_read_time_us":2086,"lbm_reads_lt_1ms":77,"lbm_write_time_us":1054,"lbm_writes_lt_1ms":27,"mutex_wait_us":136,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":54,"thread_start_us":19166,"threads_started":1}
I20260812 20:03:59.545778 18503 log.cc:1199] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/2e1ec98f1d914666bc5e02306ee16d13
I20260812 20:03:59.546506 18503 ts_tablet_manager.cc:1950] T 2e1ec98f1d914666bc5e02306ee16d13 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.549193 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 2e1ec98f1d914666bc5e02306ee16d13 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.549738 18503 tablet_replica.cc:333] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.550040 18503 raft_consensus.cc:2243] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.551010 18503 raft_consensus.cc:2272] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.552469 18503 ts_tablet_manager.cc:1916] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.552835 17755 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.555218 18501 log.cc:1199] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/aad9b9e5ec944a1aa309cc9332b2dbca
I20260812 20:03:59.538256 18515 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.559303 18515 ts_tablet_manager.cc:1403] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.212s user 0.004s sys 0.007s
I20260812 20:03:59.559566 18503 ts_tablet_manager.cc:1929] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.559806 18503 log.cc:1199] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:03:59.560385 18503 ts_tablet_manager.cc:1950] T 77032e8e8636495a9dfdfa1ce5d85021 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.560596 18515 raft_consensus.cc:359] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.560916 18515 raft_consensus.cc:385] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.561019 18515 raft_consensus.cc:740] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.561388 18515 consensus_queue.cc:260] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.562129 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 77032e8e8636495a9dfdfa1ce5d85021 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.562534 18503 tablet_replica.cc:333] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.562806 18503 raft_consensus.cc:2243] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.563072 18503 raft_consensus.cc:2272] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.563966 18515 ts_tablet_manager.cc:1434] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260812 20:03:59.564352 18503 ts_tablet_manager.cc:1916] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.564414 18515 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.569463 18501 ts_tablet_manager.cc:1950] T aad9b9e5ec944a1aa309cc9332b2dbca P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.571276 18503 ts_tablet_manager.cc:1929] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.571560 18503 log.cc:1199] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5000368644a64059af5235482fe2e79c
I20260812 20:03:59.572119 18503 ts_tablet_manager.cc:1950] T 5000368644a64059af5235482fe2e79c P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.573378 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5000368644a64059af5235482fe2e79c (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.574074 18503 tablet_replica.cc:333] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.574404 18503 raft_consensus.cc:2243] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.574806 18503 raft_consensus.cc:2272] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.575320 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet aad9b9e5ec944a1aa309cc9332b2dbca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.575996 18503 ts_tablet_manager.cc:1916] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.577505 18501 tablet_replica.cc:333] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.577912 18501 raft_consensus.cc:2243] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.583348 18503 ts_tablet_manager.cc:1929] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.583618 18503 log.cc:1199] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/7e7e3cf6c3324a42aa08da0128dda7ca
I20260812 20:03:59.584256 18503 ts_tablet_manager.cc:1950] T 7e7e3cf6c3324a42aa08da0128dda7ca P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.584887 18501 raft_consensus.cc:2272] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.585559 18515 tablet_bootstrap.cc:654] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.585700 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 7e7e3cf6c3324a42aa08da0128dda7ca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.586175 18503 tablet_replica.cc:333] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.586449 18503 raft_consensus.cc:2243] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.586707 18503 raft_consensus.cc:2272] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.587206 18501 ts_tablet_manager.cc:1916] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.587893 18503 ts_tablet_manager.cc:1916] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.595242 18503 ts_tablet_manager.cc:1929] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.595498 18503 log.cc:1199] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/9f052b8cb7864b5693be4845f8c473ac
I20260812 20:03:59.596007 18503 ts_tablet_manager.cc:1950] T 9f052b8cb7864b5693be4845f8c473ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.596750 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.154s user 0.013s sys 0.010s Metrics: {"dirs.queue_time_us":4925,"dirs.run_cpu_time_us":1032,"dirs.run_wall_time_us":8121,"lbm_write_time_us":400,"lbm_writes_lt_1ms":7,"mutex_wait_us":3392,"num_input_rowsets":0,"thread_start_us":99488,"threads_started":1}
I20260812 20:03:59.597519 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 9f052b8cb7864b5693be4845f8c473ac (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.598042 18503 tablet_replica.cc:333] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.598184 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.403328
I20260812 20:03:59.598285 18503 raft_consensus.cc:2243] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.598482 18503 raft_consensus.cc:2272] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.598912 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.403328
I20260812 20:03:59.599607 18503 ts_tablet_manager.cc:1916] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.599618 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.403328
I20260812 20:03:59.600291 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.403328
I20260812 20:03:59.606696 18503 ts_tablet_manager.cc:1929] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.606878 18503 log.cc:1199] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/6397453a07df4814be9827bb15738081
I20260812 20:03:59.607389 18503 ts_tablet_manager.cc:1950] T 6397453a07df4814be9827bb15738081 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.608851 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 6397453a07df4814be9827bb15738081 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
W20260812 20:03:59.609771 17750 catalog_manager.cc:5017] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): delete failed for tablet 6397453a07df4814be9827bb15738081 because the tablet was not found. No further retry: Not found: Tablet not found: 6397453a07df4814be9827bb15738081
I20260812 20:03:59.610116 18503 tablet_replica.cc:333] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.610332 18503 raft_consensus.cc:2243] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.610528 18503 raft_consensus.cc:2272] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.611171 18657 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.611693 18503 ts_tablet_manager.cc:1916] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.613327 18657 tablet_bootstrap.cc:654] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.617429 18633 ts_tablet_manager.cc:1929] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.620039 18633 log.cc:1199] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/c5c9d056274f4c1ab24c5f317981d016
I20260812 20:03:59.621166 18633 ts_tablet_manager.cc:1950] T c5c9d056274f4c1ab24c5f317981d016 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.623868 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet c5c9d056274f4c1ab24c5f317981d016 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.628057 18633 tablet_replica.cc:333] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.629034 18503 ts_tablet_manager.cc:1929] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.629827 18503 log.cc:1199] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/44959dd27d9642f0ad8ddf8395887083
I20260812 20:03:59.630177 18633 raft_consensus.cc:2243] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.630514 18503 ts_tablet_manager.cc:1950] T 44959dd27d9642f0ad8ddf8395887083 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:03:59.631995 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 44959dd27d9642f0ad8ddf8395887083 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.632507 18503 tablet_replica.cc:333] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:03:59.632802 18503 raft_consensus.cc:2243] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.630749 18633 raft_consensus.cc:2272] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.633447 18503 raft_consensus.cc:2272] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.634502 18633 ts_tablet_manager.cc:1916] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.634773 17758 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.635816 18503 ts_tablet_manager.cc:1916] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.639317 18508 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:59.639576 18508 ts_tablet_manager.cc:1403] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.274s user 0.014s sys 0.002s
I20260812 20:03:59.640694 18508 raft_consensus.cc:359] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.641113 18508 raft_consensus.cc:385] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.641275 18508 raft_consensus.cc:740] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.641853 18508 consensus_queue.cc:260] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.642570 18515 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.642796 18515 ts_tablet_manager.cc:1403] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.001s
I20260812 20:03:59.642908 18657 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.643225 18657 ts_tablet_manager.cc:1403] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.007s
I20260812 20:03:59.643893 18515 raft_consensus.cc:359] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.644134 18515 raft_consensus.cc:385] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.644230 18515 raft_consensus.cc:740] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.645833 18657 raft_consensus.cc:359] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.646166 18657 raft_consensus.cc:385] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.646349 18657 raft_consensus.cc:740] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.646843 18657 consensus_queue.cc:260] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.649370 18657 ts_tablet_manager.cc:1434] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 20:03:59.649916 18657 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.655447 18657 tablet_bootstrap.cc:654] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.657838 18379 raft_consensus.cc:493] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.658078 18379 raft_consensus.cc:515] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.658890 18379 leader_election.cc:290] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.660035 18503 ts_tablet_manager.cc:1929] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.660169 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.660320 18503 log.cc:1199] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/8dfafbefa0be43d694d032f733f1d7b1
I20260812 20:03:59.660317 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:59.660533 18508 ts_tablet_manager.cc:1434] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20260812 20:03:59.660871 18503 ts_tablet_manager.cc:1950] T 8dfafbefa0be43d694d032f733f1d7b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
W20260812 20:03:59.660952 17883 leader_election.cc:343] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:59.661182 17884 leader_election.cc:343] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.661336 17884 leader_election.cc:304] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:59.662025 18508 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:59.662248 18379 raft_consensus.cc:2749] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.664232 18508 tablet_bootstrap.cc:654] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.666077 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 8dfafbefa0be43d694d032f733f1d7b1 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
W20260812 20:03:59.666918 17750 catalog_manager.cc:5017] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): delete failed for tablet 77032e8e8636495a9dfdfa1ce5d85021 because the tablet was not found. No further retry: Not found: Tablet not found: 77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:03:59.666994 18633 ts_tablet_manager.cc:1929] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.667128 18633 log.cc:1199] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b607a7c7ca104dbca81ab5ddae140a58
I20260812 20:03:59.668431 18633 ts_tablet_manager.cc:1950] T b607a7c7ca104dbca81ab5ddae140a58 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.669884 18515 consensus_queue.cc:260] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.671352 18657 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.671614 18657 ts_tablet_manager.cc:1403] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260812 20:03:59.672358 18515 ts_tablet_manager.cc:1434] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.029s user 0.002s sys 0.001s
I20260812 20:03:59.672751 18515 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.672729 18657 raft_consensus.cc:359] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.673321 18657 raft_consensus.cc:385] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.673429 18657 raft_consensus.cc:740] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.673584 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b607a7c7ca104dbca81ab5ddae140a58 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.673789 18657 consensus_queue.cc:260] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.674319 18657 ts_tablet_manager.cc:1434] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:03:59.674731 18657 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.649344 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.050s user 0.007s sys 0.005s Metrics: {"dirs.queue_time_us":16580,"dirs.run_cpu_time_us":1087,"dirs.run_wall_time_us":12426,"lbm_write_time_us":390,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"spinlock_wait_cycles":4992,"thread_start_us":14200,"threads_started":1}
I20260812 20:03:59.677043 18633 tablet_replica.cc:333] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.677345 18657 tablet_bootstrap.cc:654] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.678877 18633 raft_consensus.cc:2243] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.679206 18633 raft_consensus.cc:2272] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.680686 18633 ts_tablet_manager.cc:1916] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.682603 18508 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:59.682994 18508 ts_tablet_manager.cc:1403] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20260812 20:03:59.683753 18508 raft_consensus.cc:359] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.683977 18508 raft_consensus.cc:385] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.684085 18508 raft_consensus.cc:740] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.684367 18508 consensus_queue.cc:260] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.684908 18508 ts_tablet_manager.cc:1434] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:59.685331 18508 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:59.687381 18508 tablet_bootstrap.cc:654] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.693811 18515 tablet_bootstrap.cc:654] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.697512 18633 ts_tablet_manager.cc:1929] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.697780 18501 ts_tablet_manager.cc:1929] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.697832 18633 log.cc:1199] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/2eb8accc87ef4ee4a5186719c042db42
I20260812 20:03:59.698081 18501 log.cc:1199] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/489ea23ba78a47bda493198355f9b340
I20260812 20:03:59.698344 18633 ts_tablet_manager.cc:1950] T 2eb8accc87ef4ee4a5186719c042db42 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.698802 18501 ts_tablet_manager.cc:1950] T 489ea23ba78a47bda493198355f9b340 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.700503 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 489ea23ba78a47bda493198355f9b340 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.702199 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 2eb8accc87ef4ee4a5186719c042db42 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.702807 18633 tablet_replica.cc:333] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.703114 18633 raft_consensus.cc:2243] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.703409 18633 pending_rounds.cc:70] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Trying to abort 1 pending ops.
I20260812 20:03:59.703632 18633 pending_rounds.cc:77] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770398584532992 op_type: NO_OP noop_request { }
I20260812 20:03:59.703905 18633 raft_consensus.cc:2889] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:03:59.704173 18633 raft_consensus.cc:2272] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.712016 18633 ts_tablet_manager.cc:1916] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.721994 18501 tablet_replica.cc:333] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.722301 18501 raft_consensus.cc:2243] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.722527 18501 raft_consensus.cc:2272] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.723428 18501 ts_tablet_manager.cc:1916] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.728528 18633 ts_tablet_manager.cc:1929] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.728725 18633 log.cc:1199] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/2e1ec98f1d914666bc5e02306ee16d13
I20260812 20:03:59.729151 18633 ts_tablet_manager.cc:1950] T 2e1ec98f1d914666bc5e02306ee16d13 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.730432 18657 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.730594 18657 ts_tablet_manager.cc:1403] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.012s
I20260812 20:03:59.730798 18633 tablet_replica.cc:333] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.731007 18633 raft_consensus.cc:2243] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.731213 18633 raft_consensus.cc:2272] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.731503 18657 raft_consensus.cc:359] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.731742 18657 raft_consensus.cc:385] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.731858 18657 raft_consensus.cc:740] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.732116 18657 consensus_queue.cc:260] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.732609 18633 ts_tablet_manager.cc:1916] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.732722 18657 ts_tablet_manager.cc:1434] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:03:59.733042 18657 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.735307 18657 tablet_bootstrap.cc:654] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:59.737578 18207 consensus_peers.cc:597] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f052b8cb7864b5693be4845f8c473ac. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:59.740154 18633 ts_tablet_manager.cc:1929] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.740324 18633 log.cc:1199] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5000368644a64059af5235482fe2e79c
I20260812 20:03:59.740753 18633 ts_tablet_manager.cc:1950] T 5000368644a64059af5235482fe2e79c P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.741683 18633 tablet_replica.cc:333] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.741940 18633 raft_consensus.cc:2243] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.742190 18633 pending_rounds.cc:70] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Trying to abort 1 pending ops.
I20260812 20:03:59.742358 18633 pending_rounds.cc:77] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770398625243136 op_type: NO_OP noop_request { }
I20260812 20:03:59.742539 18633 raft_consensus.cc:2889] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:03:59.742722 18633 raft_consensus.cc:2272] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.742920 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 2e1ec98f1d914666bc5e02306ee16d13 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.743316 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5000368644a64059af5235482fe2e79c (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.743836 18633 ts_tablet_manager.cc:1916] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.747049 17757 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.749475 18633 ts_tablet_manager.cc:1929] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.749782 18633 log.cc:1199] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/7e7e3cf6c3324a42aa08da0128dda7ca
I20260812 20:03:59.750346 18633 ts_tablet_manager.cc:1950] T 7e7e3cf6c3324a42aa08da0128dda7ca P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.751657 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 7e7e3cf6c3324a42aa08da0128dda7ca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.752130 18633 tablet_replica.cc:333] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.752408 18633 raft_consensus.cc:2243] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.752686 18657 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.752660 18633 raft_consensus.cc:2272] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.752908 18657 ts_tablet_manager.cc:1403] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260812 20:03:59.753222 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.145s user 0.010s sys 0.042s Metrics: {"dirs.queue_time_us":1028,"dirs.run_cpu_time_us":1017,"dirs.run_wall_time_us":5653,"lbm_write_time_us":452,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"spinlock_wait_cycles":45184}
I20260812 20:03:59.754016 18633 ts_tablet_manager.cc:1916] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.753927 18657 raft_consensus.cc:359] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.754149 18657 raft_consensus.cc:385] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.754235 18657 raft_consensus.cc:740] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.754541 18657 consensus_queue.cc:260] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.755138 18657 ts_tablet_manager.cc:1434] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:03:59.755532 18657 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.758024 18515 tablet_bootstrap.cc:492] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.758164 18657 tablet_bootstrap.cc:654] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.758285 18515 ts_tablet_manager.cc:1403] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.004s
I20260812 20:03:59.759174 18515 raft_consensus.cc:359] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.759426 18515 raft_consensus.cc:385] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.759554 18515 raft_consensus.cc:740] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.759967 18515 consensus_queue.cc:260] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.760641 18515 ts_tablet_manager.cc:1434] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:59.761029 18515 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.763000 18515 tablet_bootstrap.cc:654] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.763041 18633 ts_tablet_manager.cc:1929] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.763355 18633 log.cc:1199] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/6397453a07df4814be9827bb15738081
I20260812 20:03:59.764017 18633 ts_tablet_manager.cc:1950] T 6397453a07df4814be9827bb15738081 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.765576 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 6397453a07df4814be9827bb15738081 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
W20260812 20:03:59.766819 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet 6397453a07df4814be9827bb15738081 because the tablet was not found. No further retry: Not found: Tablet not found: 6397453a07df4814be9827bb15738081
I20260812 20:03:59.767391 18633 tablet_replica.cc:333] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.767741 18633 raft_consensus.cc:2243] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.768070 18633 raft_consensus.cc:2272] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.769155 18501 ts_tablet_manager.cc:1929] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:03:59.769955 18633 ts_tablet_manager.cc:1916] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.776582 18633 ts_tablet_manager.cc:1929] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.776820 18633 log.cc:1199] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:03:59.777305 18633 ts_tablet_manager.cc:1950] T 77032e8e8636495a9dfdfa1ce5d85021 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.778450 18633 tablet_replica.cc:333] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.778759 18633 raft_consensus.cc:2243] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.778899 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 77032e8e8636495a9dfdfa1ce5d85021 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.779197 18633 raft_consensus.cc:2272] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.780025 18508 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:59.780211 18508 ts_tablet_manager.cc:1403] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.017s
I20260812 20:03:59.780295 18633 ts_tablet_manager.cc:1916] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.781172 18508 raft_consensus.cc:359] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.781384 18508 raft_consensus.cc:385] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.781477 18508 raft_consensus.cc:740] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.781805 18508 consensus_queue.cc:260] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.782382 18508 ts_tablet_manager.cc:1434] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:59.785037 18508 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:59.788842 18508 tablet_bootstrap.cc:654] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.789460 18633 ts_tablet_manager.cc:1929] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.789708 18633 log.cc:1199] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/44959dd27d9642f0ad8ddf8395887083
I20260812 20:03:59.790530 18633 ts_tablet_manager.cc:1950] T 44959dd27d9642f0ad8ddf8395887083 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.792022 18633 tablet_replica.cc:333] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.792203 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 44959dd27d9642f0ad8ddf8395887083 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.792307 18633 raft_consensus.cc:2243] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 20:03:59.779685 18043 consensus_peers.cc:597] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6397453a07df4814be9827bb15738081. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:59.792551 18633 raft_consensus.cc:2272] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.781809 18515 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.792831 18657 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.792979 18515 ts_tablet_manager.cc:1403] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20260812 20:03:59.793040 18657 ts_tablet_manager.cc:1403] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.002s
I20260812 20:03:59.793910 18633 ts_tablet_manager.cc:1916] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.794164 18515 raft_consensus.cc:359] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.794229 18657 raft_consensus.cc:359] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.794402 18515 raft_consensus.cc:385] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.794463 18657 raft_consensus.cc:385] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.794518 18515 raft_consensus.cc:740] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.794569 18657 raft_consensus.cc:740] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.794879 18515 consensus_queue.cc:260] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.794859 18657 consensus_queue.cc:260] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.795512 18657 ts_tablet_manager.cc:1434] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:59.799363 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2852 bytes on disk
I20260812 20:03:59.799747 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2852 bytes on disk
I20260812 20:03:59.800640 18515 ts_tablet_manager.cc:1434] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260812 20:03:59.800582 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20260812 20:03:59.801061 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:03:59.802062 18633 ts_tablet_manager.cc:1929] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.802295 18633 log.cc:1199] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/8dfafbefa0be43d694d032f733f1d7b1
I20260812 20:03:59.802955 18633 ts_tablet_manager.cc:1950] T 8dfafbefa0be43d694d032f733f1d7b1 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.805294 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 8dfafbefa0be43d694d032f733f1d7b1 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.805789 18379 raft_consensus.cc:493] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.805862 18633 tablet_replica.cc:333] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:03:59.806006 18379 raft_consensus.cc:515] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.806306 18633 raft_consensus.cc:2243] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:03:59.806751 18657 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.806846 18379 leader_election.cc:290] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.807516 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd14ad61b704e56a9dc1f37ce97925c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.807790 18125 raft_consensus.cc:2468] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:59.807950 18515 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.808480 17884 leader_election.cc:304] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:59.808885 18379 raft_consensus.cc:2804] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:59.809018 18379 raft_consensus.cc:493] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:59.809108 18379 raft_consensus.cc:3060] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.809906 18501 log.cc:1199] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/c5c9d056274f4c1ab24c5f317981d016
I20260812 20:03:59.810438 18501 ts_tablet_manager.cc:1950] T c5c9d056274f4c1ab24c5f317981d016 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.810734 18515 tablet_bootstrap.cc:654] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.811681 18501 tablet_replica.cc:333] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.811909 18501 raft_consensus.cc:2243] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.812155 18501 raft_consensus.cc:2272] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.812322 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet c5c9d056274f4c1ab24c5f317981d016 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.812788 18379 raft_consensus.cc:515] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
W20260812 20:03:59.818068 18043 consensus_peers.cc:597] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6397453a07df4814be9827bb15738081. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:03:59.818506 18379 leader_election.cc:290] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.818717 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd14ad61b704e56a9dc1f37ce97925c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:03:59.818994 18125 raft_consensus.cc:3060] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.818158 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.218s user 0.033s sys 0.008s Metrics: {"bytes_written":25078,"cfile_cache_hit":16,"cfile_cache_hit_bytes":18896,"cfile_cache_miss":40,"cfile_cache_miss_bytes":27300,"cfile_init":11,"delta_iterators_relevant":12,"dirs.queue_time_us":2368,"dirs.run_cpu_time_us":1765,"dirs.run_wall_time_us":10741,"drs_written":1,"lbm_read_time_us":38740,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":82,"lbm_write_time_us":1160,"lbm_writes_lt_1ms":27,"mutex_wait_us":132,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":54,"spinlock_wait_cycles":12160,"thread_start_us":15830,"threads_started":1}
I20260812 20:03:59.822355 18125 raft_consensus.cc:2468] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:59.822973 17884 leader_election.cc:304] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:03:59.823335 18379 raft_consensus.cc:2804] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:59.823469 18379 raft_consensus.cc:697] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:59.823765 18379 consensus_queue.cc:237] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.827521 18633 raft_consensus.cc:2272] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.830335 18633 ts_tablet_manager.cc:1916] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.830569 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd14ad61b704e56a9dc1f37ce97925c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:59.832384 17769 catalog_manager.cc:5719] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260812 20:03:59.834385 18501 ts_tablet_manager.cc:1916] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:03:59.835311 17883 leader_election.cc:343] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.837296 18633 ts_tablet_manager.cc:1929] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.837495 18633 log.cc:1199] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/9f052b8cb7864b5693be4845f8c473ac
I20260812 20:03:59.837690 18657 tablet_bootstrap.cc:654] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.838919 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bd14ad61b704e56a9dc1f37ce97925c" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:59.839951 17883 leader_election.cc:343] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.841862 18633 ts_tablet_manager.cc:1950] T 9f052b8cb7864b5693be4845f8c473ac P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:03:59.843721 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 9f052b8cb7864b5693be4845f8c473ac (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.856848 18501 ts_tablet_manager.cc:1929] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.858793 18501 log.cc:1199] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b607a7c7ca104dbca81ab5ddae140a58
I20260812 20:03:59.861845 18657 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.860227 18515 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.862129 18657 ts_tablet_manager.cc:1403] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.005s
I20260812 20:03:59.862228 18515 ts_tablet_manager.cc:1403] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.000s
I20260812 20:03:59.863296 18515 raft_consensus.cc:359] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.863314 18657 raft_consensus.cc:359] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.863590 18515 raft_consensus.cc:385] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.863633 18657 raft_consensus.cc:385] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.863685 18515 raft_consensus.cc:740] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.863726 18657 raft_consensus.cc:740] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.864027 18515 consensus_queue.cc:260] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.864073 18657 consensus_queue.cc:260] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.864732 18515 ts_tablet_manager.cc:1434] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:59.864852 18657 ts_tablet_manager.cc:1434] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260812 20:03:59.865278 18515 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.865336 18657 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.867467 18515 tablet_bootstrap.cc:654] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.867967 18657 tablet_bootstrap.cc:654] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.873383 18501 ts_tablet_manager.cc:1950] T b607a7c7ca104dbca81ab5ddae140a58 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.877347 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b607a7c7ca104dbca81ab5ddae140a58 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.877940 18501 tablet_replica.cc:333] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.878305 18501 raft_consensus.cc:2243] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.878597 18501 raft_consensus.cc:2272] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.882731 18508 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:03:59.882964 18508 ts_tablet_manager.cc:1403] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.001s
I20260812 20:03:59.884032 18508 raft_consensus.cc:359] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.884296 18508 raft_consensus.cc:385] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.884300 17755 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:03:59.884380 18508 raft_consensus.cc:740] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.884756 18508 consensus_queue.cc:260] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.885452 18508 ts_tablet_manager.cc:1434] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:03:59.885812 18620 raft_consensus.cc:493] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.886016 18620 raft_consensus.cc:515] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.886854 18620 leader_election.cc:290] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.888461 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.888553 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:59.889382 18207 leader_election.cc:343] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.889963 18508 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:03:59.889768 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.240s user 0.009s sys 0.075s Metrics: {"dirs.queue_time_us":4564,"dirs.run_cpu_time_us":1131,"dirs.run_wall_time_us":5903,"lbm_write_time_us":499,"lbm_writes_lt_1ms":7,"mutex_wait_us":1702,"num_input_rowsets":0}
W20260812 20:03:59.890138 18208 leader_election.cc:343] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.890331 18208 leader_election.cc:304] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.892472 18508 tablet_bootstrap.cc:654] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.894918 18620 raft_consensus.cc:2749] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.898741 18501 ts_tablet_manager.cc:1916] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.909929 18379 raft_consensus.cc:493] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.910249 18379 raft_consensus.cc:515] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.911103 18379 leader_election.cc:290] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.912454 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.912992 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:03:59.913246 18281 raft_consensus.cc:2468] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:03:59.913903 17883 leader_election.cc:304] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
W20260812 20:03:59.914608 17884 leader_election.cc:343] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.915177 18379 raft_consensus.cc:2804] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:59.915302 18379 raft_consensus.cc:493] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:59.915429 18379 raft_consensus.cc:3060] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.924937 18501 ts_tablet_manager.cc:1929] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:03:59.925140 18501 log.cc:1199] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/2eb8accc87ef4ee4a5186719c042db42
I20260812 20:03:59.925786 18501 ts_tablet_manager.cc:1950] T 2eb8accc87ef4ee4a5186719c042db42 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:03:59.927057 18501 tablet_replica.cc:333] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:03:59.927309 18501 raft_consensus.cc:2243] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:03:59.927345 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 2eb8accc87ef4ee4a5186719c042db42 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:03:59.927582 18501 raft_consensus.cc:2272] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:03:59.927613 18379 raft_consensus.cc:515] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.929438 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
W20260812 20:03:59.930195 17884 leader_election.cc:343] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.931246 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:59.931520 18281 raft_consensus.cc:3060] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.935554 18281 raft_consensus.cc:2468] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:03:59.936371 17883 leader_election.cc:304] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:03:59.936904 18591 raft_consensus.cc:2804] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:03:59.937094 18591 raft_consensus.cc:697] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:03:59.937484 18591 consensus_queue.cc:237] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.938400 18657 tablet_bootstrap.cc:492] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:03:59.938603 18657 ts_tablet_manager.cc:1403] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.032s
I20260812 20:03:59.939539 18657 raft_consensus.cc:359] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.939749 18657 raft_consensus.cc:385] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.939844 18657 raft_consensus.cc:740] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.940124 18657 consensus_queue.cc:260] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:03:59.940939 18657 ts_tablet_manager.cc:1434] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:03:59.941278 18657 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:03:59.941798 18379 leader_election.cc:290] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.944155 18515 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:03:59.944336 18515 ts_tablet_manager.cc:1403] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.007s
I20260812 20:03:59.945235 18515 raft_consensus.cc:359] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.945468 18515 raft_consensus.cc:385] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:03:59.945571 18515 raft_consensus.cc:740] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:03:59.945869 18515 consensus_queue.cc:260] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.946517 18515 ts_tablet_manager.cc:1434] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:03:59.947124 18501 ts_tablet_manager.cc:1916] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:03:59.948196 18515 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:03:59.948807 18620 raft_consensus.cc:493] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.949009 18620 raft_consensus.cc:515] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.949800 18620 leader_election.cc:290] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.950378 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.950785 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:03:59.951270 18207 leader_election.cc:343] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.951541 18515 tablet_bootstrap.cc:654] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
W20260812 20:03:59.951519 18208 leader_election.cc:343] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.951694 18208 leader_election.cc:304] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.952518 18620 raft_consensus.cc:2749] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.953725 18657 tablet_bootstrap.cc:654] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:03:59.968324 18406 raft_consensus.cc:493] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.968566 18406 raft_consensus.cc:515] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.969455 18406 leader_election.cc:290] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.970000 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.970038 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:59.970675 18043 leader_election.cc:343] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:03:59.970916 18043 leader_election.cc:343] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.971043 18043 leader_election.cc:304] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:03:59.971352 18406 raft_consensus.cc:2749] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:03:59.986012 18620 raft_consensus.cc:493] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.986213 18620 raft_consensus.cc:515] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.987031 18620 leader_election.cc:290] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:03:59.987731 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:03:59.987949 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.988010 18125 raft_consensus.cc:2468] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:03:59.988463 18406 raft_consensus.cc:493] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:03:59.988651 18406 raft_consensus.cc:515] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.988677 18208 leader_election.cc:304] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:59.989306 18406 leader_election.cc:290] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.989929 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:03:59.990161 18207 leader_election.cc:343] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:03:59.990254 18281 raft_consensus.cc:2468] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:59.990590 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:03:59.990821 17961 raft_consensus.cc:2468] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:03:59.991011 18043 leader_election.cc:304] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:03:59.991415 18406 raft_consensus.cc:2804] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:59.991528 18406 raft_consensus.cc:493] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:59.991632 18406 raft_consensus.cc:3060] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.992441 18620 raft_consensus.cc:2804] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:03:59.992580 18620 raft_consensus.cc:493] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:03:59.992658 18620 raft_consensus.cc:3060] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:03:59.994891 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:03:59.994835 18406 raft_consensus.cc:515] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:03:59.998371 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:03:59.998667 18406 leader_election.cc:290] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:03:59.999073 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a4e1ea1f52461c8aec59967842b390" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:03:59.999315 17961 raft_consensus.cc:3060] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.001263 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.001884 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.002063 18501 ts_tablet_manager.cc:1929] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.002188 18501 log.cc:1199] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/2e1ec98f1d914666bc5e02306ee16d13
I20260812 20:04:00.002384 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.002635 18501 ts_tablet_manager.cc:1950] T 2e1ec98f1d914666bc5e02306ee16d13 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.003672 18501 tablet_replica.cc:333] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.004194 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 2e1ec98f1d914666bc5e02306ee16d13 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.005764 18281 raft_consensus.cc:3060] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.005813 17961 raft_consensus.cc:2468] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.007310 18043 leader_election.cc:304] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.007632 18501 raft_consensus.cc:2243] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.007642 18406 raft_consensus.cc:2804] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.009227 18281 raft_consensus.cc:2468] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.009444 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.009863 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.010156 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.010444 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.011058 18620 raft_consensus.cc:515] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.012462 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.012696 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.013103 18406 raft_consensus.cc:697] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
W20260812 20:04:00.013183 18207 leader_election.cc:343] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.013182 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.013471 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.013466 18406 consensus_queue.cc:237] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.014801 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.015012 18125 raft_consensus.cc:3060] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.015383 17767 catalog_manager.cc:5719] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.016355 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.003s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.014665 18620 leader_election.cc:290] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.022684 18508 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.022877 18508 ts_tablet_manager.cc:1403] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.000s
I20260812 20:04:00.023829 18508 raft_consensus.cc:359] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.024035 18508 raft_consensus.cc:385] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.024117 18508 raft_consensus.cc:740] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.024358 18508 consensus_queue.cc:260] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.024932 18508 ts_tablet_manager.cc:1434] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:04:00.025962 18508 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.028378 18125 raft_consensus.cc:2468] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.029049 18208 leader_election.cc:304] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.031497 18501 raft_consensus.cc:2272] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.032936 18508 tablet_bootstrap.cc:654] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.035498 18501 ts_tablet_manager.cc:1916] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.036676 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.037077 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.037374 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.037493 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.037497 18620 raft_consensus.cc:2804] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.037834 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.038089 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.038350 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.038658 18620 raft_consensus.cc:697] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.039983 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.040126 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.040267 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.040441 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.040452 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.040690 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.040817 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.040547 18620 consensus_queue.cc:237] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.041085 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.041395 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.041765 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.044515 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.045091 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.045344 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.045550 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.045784 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.049037 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.049271 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.049453 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.053750 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":182,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.055513 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.006s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":61,"lbm_read_time_us":148,"lbm_reads_lt_1ms":8,"lbm_write_time_us":162,"lbm_writes_lt_1ms":4,"mutex_wait_us":4487,"reinsert_count":0,"update_count":2}
I20260812 20:04:00.056468 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.005s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2880,"lbm_read_time_us":154,"lbm_reads_lt_1ms":8,"lbm_write_time_us":181,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":2743,"threads_started":1,"update_count":2}
I20260812 20:04:00.064206 17767 catalog_manager.cc:5719] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.066620 18501 ts_tablet_manager.cc:1929] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.066860 18501 log.cc:1199] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:04:00.067341 18501 ts_tablet_manager.cc:1950] T 77032e8e8636495a9dfdfa1ce5d85021 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.068710 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 77032e8e8636495a9dfdfa1ce5d85021 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.069168 18501 tablet_replica.cc:333] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.069412 18501 raft_consensus.cc:2243] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.069598 18501 pending_rounds.cc:70] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Trying to abort 1 pending ops.
I20260812 20:04:00.069788 18501 pending_rounds.cc:77] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770398498729984 op_type: NO_OP noop_request { }
I20260812 20:04:00.069974 18501 raft_consensus.cc:2889] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:00.070093 18501 raft_consensus.cc:2272] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.071215 18501 ts_tablet_manager.cc:1916] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.073427 18591 raft_consensus.cc:493] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.073675 18591 raft_consensus.cc:515] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.074494 18591 leader_election.cc:290] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.075345 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.075608 18125 raft_consensus.cc:2468] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:00.076210 17884 leader_election.cc:304] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.076421 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.076603 18591 raft_consensus.cc:2804] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.076700 18591 raft_consensus.cc:493] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.076814 18591 raft_consensus.cc:3060] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:00.077124 17883 leader_election.cc:343] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.077234 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.077589 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.077924 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.077957 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.078190 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.078292 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.078588 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.078867 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.080009 18591 raft_consensus.cc:515] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.080235 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.080575 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.080667 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.080868 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.080986 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.081212 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.081370 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.081365 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.081598 18125 raft_consensus.cc:3060] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.081671 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.081981 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.082314 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.082309 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.082535 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260812 20:04:00.083000 17883 leader_election.cc:343] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.083217 18591 leader_election.cc:290] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.084858 18125 raft_consensus.cc:2468] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:00.085357 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.001s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.085538 17884 leader_election.cc:304] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:04:00.085760 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.086068 18591 raft_consensus.cc:2804] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.086100 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.086256 18591 raft_consensus.cc:697] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:00.086367 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.086619 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.086597 18591 consensus_queue.cc:237] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.086926 18501 ts_tablet_manager.cc:1929] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.087112 18501 log.cc:1199] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5000368644a64059af5235482fe2e79c
I20260812 20:04:00.087631 18501 ts_tablet_manager.cc:1950] T 5000368644a64059af5235482fe2e79c P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.089792 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.090054 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.090255 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.092442 17767 catalog_manager.cc:5719] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.095049 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":179,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.095631 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":705,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":180,"lbm_reads_lt_1ms":8,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260812 20:04:00.097232 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.004s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":705,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":50,"lbm_read_time_us":196,"lbm_reads_lt_1ms":8,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"mutex_wait_us":6075,"reinsert_count":0,"update_count":3}
I20260812 20:04:00.101940 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5000368644a64059af5235482fe2e79c (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.102438 18501 tablet_replica.cc:333] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.102705 18501 raft_consensus.cc:2243] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.102972 18501 raft_consensus.cc:2272] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.104358 18501 ts_tablet_manager.cc:1916] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.106703 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.107098 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.107371 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.107630 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.108556 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.108968 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.109288 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.109588 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.109648 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.109947 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.110162 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.110562 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.110837 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.111039 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.111483 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.111768 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.112008 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.113798 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.114037 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.116735 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.001s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":144,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.117146 18501 ts_tablet_manager.cc:1929] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.117309 18501 log.cc:1199] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/7e7e3cf6c3324a42aa08da0128dda7ca
I20260812 20:04:00.117673 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.117899 18501 ts_tablet_manager.cc:1950] T 7e7e3cf6c3324a42aa08da0128dda7ca P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.117972 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.118201 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.118429 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.119318 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 7e7e3cf6c3324a42aa08da0128dda7ca (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.121419 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.121955 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.122210 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.122386 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.123790 18501 tablet_replica.cc:333] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.123805 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.005s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":742,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":139,"lbm_reads_lt_1ms":8,"lbm_write_time_us":216,"lbm_writes_lt_1ms":4,"mutex_wait_us":4347,"reinsert_count":0,"update_count":4}
I20260812 20:04:00.124068 18501 raft_consensus.cc:2243] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.124310 18501 raft_consensus.cc:2272] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.126477 18501 ts_tablet_manager.cc:1916] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.128917 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.007s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":742,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":65,"lbm_read_time_us":127,"lbm_reads_lt_1ms":8,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":4}
I20260812 20:04:00.133802 17767 catalog_manager.cc:5719] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.144086 18501 ts_tablet_manager.cc:1929] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.144285 18501 log.cc:1199] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/9f052b8cb7864b5693be4845f8c473ac
I20260812 20:04:00.144763 18501 ts_tablet_manager.cc:1950] T 9f052b8cb7864b5693be4845f8c473ac P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.145161 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.145552 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.145862 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.146169 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.146247 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 9f052b8cb7864b5693be4845f8c473ac (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.148233 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.148617 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.148840 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.149415 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.149722 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.149984 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.150345 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.150583 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.150843 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.151199 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.151444 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.151669 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.152439 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":147,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.153514 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.153833 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.154007 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.154336 18591 raft_consensus.cc:493] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.154534 18591 raft_consensus.cc:515] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.154872 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.155149 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.155400 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.155444 18591 leader_election.cc:290] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.156085 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.156109 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.156299 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.156306 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.156843 18501 tablet_replica.cc:333] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
W20260812 20:04:00.156885 17883 leader_election.cc:343] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.157037 17884 leader_election.cc:343] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.157091 18501 raft_consensus.cc:2243] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:00.157191 17884 leader_election.cc:304] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.157511 18501 raft_consensus.cc:2272] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.157650 18591 raft_consensus.cc:2749] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.158701 18501 ts_tablet_manager.cc:1916] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.161479 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":775,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":58,"lbm_read_time_us":138,"lbm_reads_lt_1ms":8,"lbm_write_time_us":179,"lbm_writes_lt_1ms":4,"mutex_wait_us":2180,"reinsert_count":0,"update_count":5}
I20260812 20:04:00.166419 18501 ts_tablet_manager.cc:1929] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.166579 18501 log.cc:1199] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/6397453a07df4814be9827bb15738081
I20260812 20:04:00.167079 18501 ts_tablet_manager.cc:1950] T 6397453a07df4814be9827bb15738081 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.167781 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":179,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.168529 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 6397453a07df4814be9827bb15738081 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.168663 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":775,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":806,"lbm_read_time_us":157,"lbm_reads_lt_1ms":8,"lbm_write_time_us":192,"lbm_writes_lt_1ms":4,"mutex_wait_us":4806,"reinsert_count":0,"update_count":5}
I20260812 20:04:00.181051 18657 tablet_bootstrap.cc:492] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.181075 18508 tablet_bootstrap.cc:492] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.181267 18508 ts_tablet_manager.cc:1403] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.155s user 0.005s sys 0.001s
I20260812 20:04:00.181268 18657 ts_tablet_manager.cc:1403] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.240s user 0.006s sys 0.002s
I20260812 20:04:00.182210 18508 raft_consensus.cc:359] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.182359 18657 raft_consensus.cc:359] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.182477 18508 raft_consensus.cc:385] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.182571 18657 raft_consensus.cc:385] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260812 20:04:00.182542 17751 catalog_manager.cc:5017] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): delete failed for tablet 6397453a07df4814be9827bb15738081 because the tablet was not found. No further retry: Not found: Tablet not found: 6397453a07df4814be9827bb15738081
I20260812 20:04:00.182570 18508 raft_consensus.cc:740] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.182657 18657 raft_consensus.cc:740] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.182869 18508 consensus_queue.cc:260] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.182930 18657 consensus_queue.cc:260] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:04:00.183377 17751 catalog_manager.cc:5017] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): delete failed for tablet 77032e8e8636495a9dfdfa1ce5d85021 because the tablet was not found. No further retry: Not found: Tablet not found: 77032e8e8636495a9dfdfa1ce5d85021
I20260812 20:04:00.183471 18657 ts_tablet_manager.cc:1434] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:04:00.183480 18508 ts_tablet_manager.cc:1434] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:04:00.183718 18501 tablet_replica.cc:333] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.183810 18657 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.183825 18508 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.183974 18501 raft_consensus.cc:2243] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.184172 18501 raft_consensus.cc:2272] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.186249 18657 tablet_bootstrap.cc:654] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.186689 18501 ts_tablet_manager.cc:1916] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.190541 18508 tablet_bootstrap.cc:654] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.190542 18515 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.190757 18515 ts_tablet_manager.cc:1403] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.243s user 0.003s sys 0.006s
I20260812 20:04:00.191188 18620 raft_consensus.cc:493] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.191403 18620 raft_consensus.cc:515] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.191681 18515 raft_consensus.cc:359] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.191903 18515 raft_consensus.cc:385] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.191987 18515 raft_consensus.cc:740] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.192160 18620 leader_election.cc:290] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.192270 18515 consensus_queue.cc:260] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.192844 18515 ts_tablet_manager.cc:1434] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:04:00.193225 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.193521 18125 raft_consensus.cc:2393] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:04:00.194660 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18b5f38beca5449ebd3f8bcb0f0284d6" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.193228 18515 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.196470 18207 leader_election.cc:304] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.197575 18501 ts_tablet_manager.cc:1929] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.197772 18501 log.cc:1199] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/8dfafbefa0be43d694d032f733f1d7b1
I20260812 20:04:00.198401 18501 ts_tablet_manager.cc:1950] T 8dfafbefa0be43d694d032f733f1d7b1 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.198916 18515 tablet_bootstrap.cc:654] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.199637 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 8dfafbefa0be43d694d032f733f1d7b1 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.199783 18620 raft_consensus.cc:3060] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.200131 18501 tablet_replica.cc:333] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:00.200388 18501 raft_consensus.cc:2243] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:00.200677 18501 raft_consensus.cc:2272] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:00.201485 18501 ts_tablet_manager.cc:1916] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:00.203663 18620 raft_consensus.cc:2749] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.213478 18657 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.213757 18657 ts_tablet_manager.cc:1403] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20260812 20:04:00.214656 18515 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.214864 18515 ts_tablet_manager.cc:1403] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20260812 20:04:00.214790 18657 raft_consensus.cc:359] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.215013 18657 raft_consensus.cc:385] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.215139 18657 raft_consensus.cc:740] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.215436 18657 consensus_queue.cc:260] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.215816 18515 raft_consensus.cc:359] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.216045 18515 raft_consensus.cc:385] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.216070 18657 ts_tablet_manager.cc:1434] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:00.216150 18515 raft_consensus.cc:740] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.216509 18657 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.216427 18515 consensus_queue.cc:260] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.217022 18515 ts_tablet_manager.cc:1434] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260812 20:04:00.217361 18515 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.219049 18657 tablet_bootstrap.cc:654] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.219869 18515 tablet_bootstrap.cc:654] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.223359 18508 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.223546 18508 ts_tablet_manager.cc:1403] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.004s
I20260812 20:04:00.224408 18508 raft_consensus.cc:359] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.224614 18508 raft_consensus.cc:385] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.224709 18508 raft_consensus.cc:740] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.225040 18508 consensus_queue.cc:260] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.225565 18508 ts_tablet_manager.cc:1434] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:04:00.225945 18508 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.228629 18508 tablet_bootstrap.cc:654] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.230482 18501 ts_tablet_manager.cc:1929] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:00.230633 18501 log.cc:1199] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/44959dd27d9642f0ad8ddf8395887083
I20260812 20:04:00.231144 18501 ts_tablet_manager.cc:1950] T 44959dd27d9642f0ad8ddf8395887083 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:00.234491 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 44959dd27d9642f0ad8ddf8395887083 (table table-1 [id=b428b54d8c7e45a1826c075891c20732]) successfully deleted
I20260812 20:04:00.239039 18515 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.239274 18515 ts_tablet_manager.cc:1403] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20260812 20:04:00.240041 18508 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.240269 18508 ts_tablet_manager.cc:1403] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260812 20:04:00.240152 18515 raft_consensus.cc:359] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.240382 18515 raft_consensus.cc:385] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.240451 18515 raft_consensus.cc:740] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.240758 18515 consensus_queue.cc:260] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.241262 18508 raft_consensus.cc:359] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.241482 18508 raft_consensus.cc:385] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.241550 18508 raft_consensus.cc:740] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.241936 18508 consensus_queue.cc:260] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.242483 18508 ts_tablet_manager.cc:1434] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:04:00.242815 18508 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.245277 18508 tablet_bootstrap.cc:654] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.247498 18591 consensus_queue.cc:1048] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.248318 18515 ts_tablet_manager.cc:1434] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260812 20:04:00.248476 18657 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.248566 18508 tablet_bootstrap.cc:492] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.248698 18508 ts_tablet_manager.cc:1403] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 20:04:00.248742 18515 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.248687 18657 ts_tablet_manager.cc:1403] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.003s
I20260812 20:04:00.249678 18508 raft_consensus.cc:359] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.249888 18508 raft_consensus.cc:385] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.249991 18508 raft_consensus.cc:740] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.249971 18657 raft_consensus.cc:359] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.250187 18657 raft_consensus.cc:385] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.250274 18657 raft_consensus.cc:740] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.250236 18508 consensus_queue.cc:260] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.250526 18657 consensus_queue.cc:260] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.250800 18508 ts_tablet_manager.cc:1434] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 20:04:00.251103 18508 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.251082 18657 ts_tablet_manager.cc:1434] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 20:04:00.251035 18567 raft_consensus.cc:493] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.251291 18567 raft_consensus.cc:515] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.251454 18657 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.252275 18567 leader_election.cc:290] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.253445 18657 tablet_bootstrap.cc:654] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.253401 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.253509 18508 tablet_bootstrap.cc:654] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.253782 18127 raft_consensus.cc:2468] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.254310 18515 tablet_bootstrap.cc:654] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.254426 18208 leader_election.cc:304] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.259470 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.261536 18567 raft_consensus.cc:2804] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.261786 18567 raft_consensus.cc:493] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.261915 18567 raft_consensus.cc:3060] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.263145 18515 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.263423 18515 ts_tablet_manager.cc:1403] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20260812 20:04:00.264631 18515 raft_consensus.cc:359] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.264901 18515 raft_consensus.cc:385] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.265036 18515 raft_consensus.cc:740] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.265442 18515 consensus_queue.cc:260] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.266096 18591 raft_consensus.cc:493] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.266222 18515 ts_tablet_manager.cc:1434] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 20:04:00.266397 18591 raft_consensus.cc:515] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.266618 18515 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.267436 18591 leader_election.cc:290] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.268554 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.269474 18515 tablet_bootstrap.cc:654] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.272182 18657 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.272449 18657 ts_tablet_manager.cc:1403] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.006s
W20260812 20:04:00.272779 18207 leader_election.cc:343] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.273803 18657 raft_consensus.cc:359] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.274118 18657 raft_consensus.cc:385] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.274232 18657 raft_consensus.cc:740] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.274575 18657 consensus_queue.cc:260] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.275272 18657 ts_tablet_manager.cc:1434] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:04:00.275735 18657 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.278091 18406 raft_consensus.cc:493] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.278349 18406 raft_consensus.cc:515] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.278777 18657 tablet_bootstrap.cc:654] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.279455 18406 leader_election.cc:290] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.281395 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06b17a7a289a449397777c720ca2af7b" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.281801 18127 raft_consensus.cc:2393] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:00.282806 18508 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.283075 18508 ts_tablet_manager.cc:1403] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.008s
I20260812 20:04:00.284271 18508 raft_consensus.cc:359] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.284574 18508 raft_consensus.cc:385] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.284672 18508 raft_consensus.cc:740] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.285073 18508 consensus_queue.cc:260] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.285943 18508 ts_tablet_manager.cc:1434] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 20:04:00.286422 18508 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.289196 18508 tablet_bootstrap.cc:654] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.295336 18379 raft_consensus.cc:493] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.295639 18379 raft_consensus.cc:515] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.296715 18379 leader_election.cc:290] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.297729 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:04:00.299964 17883 leader_election.cc:343] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.308107 18281 raft_consensus.cc:3060] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.316824 18591 consensus_queue.cc:1048] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.321334 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.322376 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.322721 18127 raft_consensus.cc:2468] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:00.323340 17884 leader_election.cc:304] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.324445 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.331872 18406 raft_consensus.cc:493] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.332134 18406 raft_consensus.cc:515] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.333086 18406 leader_election.cc:290] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.334443 18591 raft_consensus.cc:3060] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.344233 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.345703 17884 leader_election.cc:304] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:04:00.346184 18379 raft_consensus.cc:2804] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.346298 18379 raft_consensus.cc:493] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.346443 18379 raft_consensus.cc:3060] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.350248 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.353148 18379 raft_consensus.cc:515] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.365224 18406 raft_consensus.cc:493] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.365594 18406 raft_consensus.cc:515] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.366745 18406 leader_election.cc:290] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
W20260812 20:04:00.368453 18043 leader_election.cc:343] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.368798 18043 leader_election.cc:343] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.369024 18043 leader_election.cc:343] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.369171 18043 leader_election.cc:304] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
W20260812 20:04:00.369875 18043 leader_election.cc:343] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.370061 18043 leader_election.cc:304] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.370494 18481 raft_consensus.cc:2749] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.370832 18481 raft_consensus.cc:2749] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.371331 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.371696 18279 raft_consensus.cc:2468] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:00.374960 18620 raft_consensus.cc:493] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.375221 18620 raft_consensus.cc:515] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.376268 18620 leader_election.cc:290] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.377416 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:00.378391 18208 leader_election.cc:343] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.383173 18567 raft_consensus.cc:515] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.380118 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.384403 17961 raft_consensus.cc:2468] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:00.383986 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.384827 17964 raft_consensus.cc:2468] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.386127 18379 leader_election.cc:290] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.387283 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.388217 18207 leader_election.cc:304] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.398412 18620 raft_consensus.cc:2804] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.398628 18620 raft_consensus.cc:493] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.398741 18620 raft_consensus.cc:3060] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.402726 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.403117 18127 raft_consensus.cc:3060] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.408305 18127 raft_consensus.cc:2468] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.416380 18567 leader_election.cc:290] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.409682 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
W20260812 20:04:00.397707 17883 leader_election.cc:343] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.417945 17884 consensus_peers.cc:597] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260812 20:04:00.425374 18207 leader_election.cc:343] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.396087 18591 raft_consensus.cc:2749] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.429333 18591 raft_consensus.cc:493] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.417023 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.431035 18126 raft_consensus.cc:3060] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.435712 18126 raft_consensus.cc:2468] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:00.419636 18043 leader_election.cc:304] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.437285 18481 raft_consensus.cc:2804] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.437454 18481 raft_consensus.cc:493] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.424383 18406 raft_consensus.cc:493] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.437599 18481 raft_consensus.cc:3060] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.441798 18406 raft_consensus.cc:515] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.442956 18406 leader_election.cc:290] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.443521 18481 raft_consensus.cc:515] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.444476 18481 leader_election.cc:290] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.447019 18208 leader_election.cc:304] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.447930 18567 raft_consensus.cc:2804] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.450452 18620 raft_consensus.cc:515] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.452469 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.452800 17964 raft_consensus.cc:3060] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.458055 17964 raft_consensus.cc:2468] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.459143 18207 leader_election.cc:304] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.462543 18591 raft_consensus.cc:515] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.463769 18591 leader_election.cc:290] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.466356 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.466764 18279 raft_consensus.cc:2468] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:00.468179 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.468519 18279 raft_consensus.cc:3060] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.473829 18279 raft_consensus.cc:2468] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.475329 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b30713b37a9473395eb907e16e77c7c" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.475699 17961 raft_consensus.cc:3060] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.480660 17961 raft_consensus.cc:2468] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.481379 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.482378 17884 leader_election.cc:304] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:04:00.488849 18481 consensus_queue.cc:1048] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.489835 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.493539 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
W20260812 20:04:00.494484 18208 leader_election.cc:343] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.495157 18591 raft_consensus.cc:2804] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.495400 18591 raft_consensus.cc:697] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:00.495945 18591 consensus_queue.cc:237] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.501734 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.504824 18716 raft_consensus.cc:2804] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.505482 18716 raft_consensus.cc:697] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.505985 18716 consensus_queue.cc:237] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.508006 18567 raft_consensus.cc:697] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.508492 18567 consensus_queue.cc:237] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.512687 18620 leader_election.cc:290] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.513501 18043 leader_election.cc:304] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.514232 18481 raft_consensus.cc:2804] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.514580 18481 raft_consensus.cc:697] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:00.515012 18481 consensus_queue.cc:237] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:04:00.517455 18207 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.14.1:46337, user_credentials={real_user=slave}} blocked reactor thread for 58382.1us
W20260812 20:04:00.519019 17883 leader_election.cc:343] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.520478 18471 consensus_queue.cc:1048] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.521322 18620 raft_consensus.cc:493] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.521667 18620 raft_consensus.cc:515] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.524000 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aac4ecab28843558e26507a083afd4e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.524406 18127 raft_consensus.cc:2468] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.525336 18208 leader_election.cc:304] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.526871 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aac4ecab28843558e26507a083afd4e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.527945 18717 raft_consensus.cc:2804] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.528142 18717 raft_consensus.cc:493] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.528347 18717 raft_consensus.cc:3060] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:00.530041 18207 leader_election.cc:343] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.530988 18043 leader_election.cc:343] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.532164 18043 leader_election.cc:304] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
W20260812 20:04:00.532979 17884 leader_election.cc:343] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.533406 17884 leader_election.cc:304] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.538964 18713 raft_consensus.cc:2749] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.547259 18657 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.547585 18657 ts_tablet_manager.cc:1403] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.272s user 0.077s sys 0.131s
I20260812 20:04:00.547928 18508 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.548125 18508 ts_tablet_manager.cc:1403] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.262s user 0.022s sys 0.088s
I20260812 20:04:00.548715 18620 leader_election.cc:290] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.548799 18657 raft_consensus.cc:359] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.549103 18657 raft_consensus.cc:385] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.549013 18508 raft_consensus.cc:359] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.549219 18508 raft_consensus.cc:385] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.549229 18657 raft_consensus.cc:740] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.549330 18508 raft_consensus.cc:740] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.549608 18508 consensus_queue.cc:260] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.549643 18657 consensus_queue.cc:260] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.550225 18657 ts_tablet_manager.cc:1434] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:04:00.550278 18508 ts_tablet_manager.cc:1434] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 20:04:00.550619 18508 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.550637 18657 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.552872 18508 tablet_bootstrap.cc:654] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.552865 18657 tablet_bootstrap.cc:654] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.554663 18717 raft_consensus.cc:515] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.555886 17767 catalog_manager.cc:5719] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.556710 18717 leader_election.cc:290] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.564688 18508 tablet_bootstrap.cc:492] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.564992 18508 ts_tablet_manager.cc:1403] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20260812 20:04:00.565481 18717 raft_consensus.cc:493] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.565912 18717 raft_consensus.cc:515] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.566629 18508 raft_consensus.cc:359] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.567380 18591 raft_consensus.cc:493] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.567416 18508 raft_consensus.cc:385] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.567523 18508 raft_consensus.cc:740] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.567593 18591 raft_consensus.cc:515] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.567975 18508 consensus_queue.cc:260] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.568411 18591 leader_election.cc:290] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.569029 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:04:00.569851 17883 leader_election.cc:343] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.554898 18515 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.570502 18515 ts_tablet_manager.cc:1403] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.304s user 0.055s sys 0.067s
I20260812 20:04:00.570688 18508 ts_tablet_manager.cc:1434] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260812 20:04:00.571199 18508 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.571492 18515 raft_consensus.cc:359] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.571729 18515 raft_consensus.cc:385] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.571828 18515 raft_consensus.cc:740] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.572110 18515 consensus_queue.cc:260] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.572631 18515 ts_tablet_manager.cc:1434] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:00.572963 18515 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.572962 18717 leader_election.cc:290] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.574169 18508 tablet_bootstrap.cc:654] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.575304 18717 consensus_queue.cc:1048] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.576769 18515 tablet_bootstrap.cc:654] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.577260 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.577422 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.577696 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.577746 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.577991 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.578050 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.578265 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.578343 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.580436 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.580610 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.580713 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.580892 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.580940 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.581207 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.578838 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.581390 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.581605 18127 raft_consensus.cc:2468] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.581732 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.582028 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.582244 18208 leader_election.cc:304] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.582373 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.582535 18657 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.582612 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.582719 18657 ts_tablet_manager.cc:1403] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.011s
I20260812 20:04:00.582764 18717 raft_consensus.cc:2804] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.582798 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.582928 18717 raft_consensus.cc:493] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.583042 18717 raft_consensus.cc:3060] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.583155 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.583434 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.583736 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.583738 18657 raft_consensus.cc:359] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.583978 18657 raft_consensus.cc:385] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.583962 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.584079 18657 raft_consensus.cc:740] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.584362 18657 consensus_queue.cc:260] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.584900 18657 ts_tablet_manager.cc:1434] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:00.585230 18657 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.585479 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aac4ecab28843558e26507a083afd4e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.585911 17961 raft_consensus.cc:3060] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.587159 18657 tablet_bootstrap.cc:654] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.589041 17961 raft_consensus.cc:2468] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.590804 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.591388 18207 leader_election.cc:304] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.594573 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":184,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.564878 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aac4ecab28843558e26507a083afd4e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.595438 18125 raft_consensus.cc:3060] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.597694 18716 raft_consensus.cc:2804] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 20:04:00.598448 18207 leader_election.cc:343] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.598830 18125 raft_consensus.cc:2468] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.599202 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":224,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.600114 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":789,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":4665,"lbm_read_time_us":161,"lbm_reads_lt_1ms":8,"lbm_write_time_us":195,"lbm_writes_lt_1ms":4,"mutex_wait_us":4183,"reinsert_count":0,"update_count":6}
W20260812 20:04:00.601405 18043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.14.3:34495, user_credentials={real_user=slave}} blocked reactor thread for 69280.6us
I20260812 20:04:00.599709 18716 raft_consensus.cc:697] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.602603 18716 consensus_queue.cc:237] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.601598 18481 raft_consensus.cc:2804] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.603142 18481 raft_consensus.cc:493] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.604624 17767 catalog_manager.cc:5719] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.617858 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.618160 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.618373 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.621039 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":789,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":24784,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":926,"lbm_read_time_us":181,"lbm_reads_lt_1ms":8,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"mutex_wait_us":4888,"reinsert_count":0,"update_count":6}
I20260812 20:04:00.621690 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.621825 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.622100 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.622360 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.622659 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.624594 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.624893 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.624845 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":6043}
I20260812 20:04:00.625159 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.625304 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":156,"lbm_writes_lt_1ms":4,"mutex_wait_us":724,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.625811 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.625876 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":6910}
I20260812 20:04:00.626127 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.626374 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.626689 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.626955 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260812 20:04:00.634383 18718 log.cc:927] Time spent T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260812 20:04:00.604404 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.636026 18481 raft_consensus.cc:3060] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.639201 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":159,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.639575 18481 raft_consensus.cc:515] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.640450 18481 leader_election.cc:290] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.640442 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":811,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1131,"lbm_read_time_us":152,"lbm_reads_lt_1ms":8,"lbm_write_time_us":206,"lbm_writes_lt_1ms":4,"mutex_wait_us":4682,"reinsert_count":0,"update_count":7}
I20260812 20:04:00.640978 18471 consensus_queue.cc:1048] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.634007 18567 raft_consensus.cc:493] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.637022 18657 tablet_bootstrap.cc:492] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
W20260812 20:04:00.646029 17884 leader_election.cc:343] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.646287 17884 leader_election.cc:304] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.647517 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.648141 18471 raft_consensus.cc:2749] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.651681 18567 raft_consensus.cc:515] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.652463 18567 leader_election.cc:290] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.653334 18657 ts_tablet_manager.cc:1403] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.012s
I20260812 20:04:00.654538 18657 raft_consensus.cc:359] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.654858 18657 raft_consensus.cc:385] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.654943 18657 raft_consensus.cc:740] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.655383 18657 consensus_queue.cc:260] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.656028 18657 ts_tablet_manager.cc:1434] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 20:04:00.656935 18657 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.659129 18657 tablet_bootstrap.cc:654] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.659267 18717 raft_consensus.cc:515] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.661173 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.661509 18127 raft_consensus.cc:2468] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.662156 18208 leader_election.cc:304] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.662274 18733 consensus_queue.cc:1048] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.662577 18717 leader_election.cc:290] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.663728 18724 raft_consensus.cc:2804] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.663928 18724 raft_consensus.cc:493] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.664148 18724 raft_consensus.cc:3060] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.665795 17769 catalog_manager.cc:5719] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.666734 17769 catalog_manager.cc:5719] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.667313 18515 tablet_bootstrap.cc:492] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.667824 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.669721 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1404f2da764dfd8797dfaad98e954f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.669966 18281 raft_consensus.cc:3060] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.667482 18515 ts_tablet_manager.cc:1403] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.018s
I20260812 20:04:00.670465 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.048s user 0.002s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":811,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":49,"lbm_read_time_us":179,"lbm_reads_lt_1ms":8,"lbm_write_time_us":184,"lbm_writes_lt_1ms":4,"mutex_wait_us":4146,"reinsert_count":0,"update_count":7}
W20260812 20:04:00.669255 18043 leader_election.cc:343] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:00.671962 18207 leader_election.cc:343] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.672466 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.673003 18281 raft_consensus.cc:2468] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.673146 18515 raft_consensus.cc:359] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.673363 18515 raft_consensus.cc:385] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.673488 18515 raft_consensus.cc:740] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
W20260812 20:04:00.673779 18207 leader_election.cc:343] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.676003 18515 consensus_queue.cc:260] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.676649 18515 ts_tablet_manager.cc:1434] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260812 20:04:00.680347 18043 leader_election.cc:304] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.680707 18406 raft_consensus.cc:2804] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.680855 18406 raft_consensus.cc:697] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:00.681191 18406 consensus_queue.cc:237] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.681248 18567 consensus_queue.cc:1048] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.681586 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.681948 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.682220 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.682502 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.683172 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.683564 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.683871 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.684233 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.684849 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.685158 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.685382 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.686407 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.686697 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.686969 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.687513 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.687790 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.688059 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.688714 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.688934 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.690006 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.690325 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.690615 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124526
I20260812 20:04:00.691047 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.691291 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.691874 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.692294 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.693524 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.693984 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.696632 18724 raft_consensus.cc:515] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.697849 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.001s sys 0.004s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":218,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:00.702078 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.004s sys 0.002s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":214,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:00.703028 18657 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.703239 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.703364 18657 ts_tablet_manager.cc:1403] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20260812 20:04:00.704507 18657 raft_consensus.cc:359] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.707974 18657 raft_consensus.cc:385] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.708854 18657 raft_consensus.cc:740] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.709786 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.710501 18127 raft_consensus.cc:3060] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:00.709228 18207 leader_election.cc:343] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:00.711513 18657 consensus_queue.cc:260] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.713760 18515 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.715667 18657 ts_tablet_manager.cc:1434] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20260812 20:04:00.716475 17767 catalog_manager.cc:5719] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.716890 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.717125 18126 raft_consensus.cc:3060] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.717363 17767 catalog_manager.cc:5719] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.718400 18657 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:00.724311 18471 consensus_queue.cc:1048] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.724857 18126 raft_consensus.cc:2468] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.726254 18515 tablet_bootstrap.cc:654] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.734364 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.734782 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.735112 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.735818 18127 raft_consensus.cc:2468] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.725397 18208 leader_election.cc:304] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.737390 18508 tablet_bootstrap.cc:492] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.737600 18508 ts_tablet_manager.cc:1403] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.167s user 0.018s sys 0.056s
I20260812 20:04:00.737663 18717 raft_consensus.cc:2804] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.737769 18208 leader_election.cc:304] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.737856 18717 raft_consensus.cc:697] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.738466 18620 raft_consensus.cc:2804] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.738158 18717 consensus_queue.cc:237] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.738641 18620 raft_consensus.cc:697] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.738742 18508 raft_consensus.cc:359] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.738888 18620 consensus_queue.cc:237] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.738987 18508 raft_consensus.cc:385] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.739089 18508 raft_consensus.cc:740] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.739383 18508 consensus_queue.cc:260] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.740152 18508 ts_tablet_manager.cc:1434] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 20:04:00.742240 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.742712 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.742985 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.713012 18724 leader_election.cc:290] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.745499 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.745913 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.748908 18508 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.749643 18715 raft_consensus.cc:493] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.749822 18715 raft_consensus.cc:515] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.750782 18715 leader_election.cc:290] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.751852 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.756173 18508 tablet_bootstrap.cc:654] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.758170 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb5c495f458246c08c470376ab0dae29" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.758620 18127 raft_consensus.cc:2393] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:00.735812 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":17685,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":737,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1087,"lbm_read_time_us":545,"lbm_reads_lt_1ms":23,"lbm_write_time_us":410,"lbm_writes_lt_1ms":9,"mutex_wait_us":2007,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":5959,"threads_started":1,"update_count":9}
I20260812 20:04:00.759271 17884 leader_election.cc:304] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.759754 18591 raft_consensus.cc:3060] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.762939 18591 raft_consensus.cc:2749] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.762773 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.004s sys 0.001s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":215,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:00.771035 18591 raft_consensus.cc:493] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.771377 18591 raft_consensus.cc:515] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.772401 17769 catalog_manager.cc:5719] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.772673 18591 leader_election.cc:290] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.771034 18508 tablet_bootstrap.cc:492] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.773263 18508 ts_tablet_manager.cc:1403] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.006s
I20260812 20:04:00.773093 17769 catalog_manager.cc:5719] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.774459 18508 raft_consensus.cc:359] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.774816 18508 raft_consensus.cc:385] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.774911 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.774966 18508 raft_consensus.cc:740] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.775296 18508 consensus_queue.cc:260] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.775926 18279 raft_consensus.cc:2468] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:00.776527 17883 leader_election.cc:304] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.776885 18713 raft_consensus.cc:2804] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.777012 18713 raft_consensus.cc:493] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.777112 18713 raft_consensus.cc:3060] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.777529 18508 ts_tablet_manager.cc:1434] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 20:04:00.780944 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.781391 18127 raft_consensus.cc:2468] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:00.781560 18713 raft_consensus.cc:515] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.782626 18713 leader_election.cc:290] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.783201 18508 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.784647 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.785017 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.785048 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.785300 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.785287 18127 raft_consensus.cc:3060] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.785593 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.785657 18508 tablet_bootstrap.cc:654] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.787671 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.787949 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.788142 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.788457 18127 raft_consensus.cc:2468] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:00.789137 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c17c36cd2a6648c49342725074cdc96e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.789410 18279 raft_consensus.cc:3060] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.789855 17884 leader_election.cc:304] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.790225 18713 raft_consensus.cc:2804] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.790412 18713 raft_consensus.cc:697] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:00.790804 18713 consensus_queue.cc:237] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.793433 18279 raft_consensus.cc:2468] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:00.794256 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.794667 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.794857 18657 tablet_bootstrap.cc:654] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.794965 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.795320 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.795583 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.796329 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.108s user 0.029s sys 0.012s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":17685,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":650,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":1241,"lbm_read_time_us":478,"lbm_reads_lt_1ms":23,"lbm_write_time_us":388,"lbm_writes_lt_1ms":9,"mutex_wait_us":6154,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":67294,"threads_started":1,"update_count":9}
I20260812 20:04:00.798843 18515 tablet_bootstrap.cc:492] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.799075 18515 ts_tablet_manager.cc:1403] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.026s
I20260812 20:04:00.800042 18515 raft_consensus.cc:359] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.800319 18515 raft_consensus.cc:385] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.800408 18515 raft_consensus.cc:740] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.800719 18515 consensus_queue.cc:260] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.801509 18515 ts_tablet_manager.cc:1434] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:00.802011 18515 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.804896 18406 raft_consensus.cc:493] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.805124 18406 raft_consensus.cc:515] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.805958 18406 leader_election.cc:290] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.808138 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":72580}
I20260812 20:04:00.808595 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":72922}
I20260812 20:04:00.809362 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.809718 18515 tablet_bootstrap.cc:654] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.809703 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.074s user 0.006s sys 0.001s Metrics: {"bytes_written":495,"delete_count":0,"lbm_write_time_us":216,"lbm_writes_lt_1ms":4,"mutex_wait_us":60892,"reinsert_count":0,"update_count":4}
I20260812 20:04:00.810763 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8750126e2be4861b6a61b34af927f54" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.810621 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.004s sys 0.002s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":209,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:00.811751 18043 leader_election.cc:304] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.812384 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":706,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":202,"lbm_reads_lt_1ms":8,"lbm_write_time_us":216,"lbm_writes_lt_1ms":4,"mutex_wait_us":912,"reinsert_count":0,"update_count":4}
I20260812 20:04:00.813092 18406 raft_consensus.cc:2749] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.816129 18515 tablet_bootstrap.cc:492] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.816401 18515 ts_tablet_manager.cc:1403] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.007s
I20260812 20:04:00.817466 18515 raft_consensus.cc:359] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.817790 18515 raft_consensus.cc:385] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.817945 18515 raft_consensus.cc:740] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.818241 18657 tablet_bootstrap.cc:492] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:00.818324 18515 consensus_queue.cc:260] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.818495 18657 ts_tablet_manager.cc:1403] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.002s
I20260812 20:04:00.819124 18515 ts_tablet_manager.cc:1434] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:04:00.819530 18515 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:00.819537 18657 raft_consensus.cc:359] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.819878 18657 raft_consensus.cc:385] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.820020 18657 raft_consensus.cc:740] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.821995 18657 consensus_queue.cc:260] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.822727 18515 tablet_bootstrap.cc:654] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.825322 18657 ts_tablet_manager.cc:1434] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260812 20:04:00.828548 17769 catalog_manager.cc:5719] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.830507 18508 tablet_bootstrap.cc:492] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.831004 18508 ts_tablet_manager.cc:1403] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.018s
I20260812 20:04:00.832123 18508 raft_consensus.cc:359] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.832373 18508 raft_consensus.cc:385] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.832502 18508 raft_consensus.cc:740] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.832893 18508 consensus_queue.cc:260] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.833256 18515 tablet_bootstrap.cc:492] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:00.833845 18508 ts_tablet_manager.cc:1434] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 20:04:00.833976 18515 ts_tablet_manager.cc:1403] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260812 20:04:00.834214 18508 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:00.834884 18515 raft_consensus.cc:359] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.835249 18515 raft_consensus.cc:385] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.835386 18515 raft_consensus.cc:740] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.835752 18515 consensus_queue.cc:260] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.836269 18508 tablet_bootstrap.cc:654] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:00.836632 18515 ts_tablet_manager.cc:1434] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 20:04:00.841922 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.842334 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.842643 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.842809 18508 tablet_bootstrap.cc:492] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:00.842959 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.843025 18508 ts_tablet_manager.cc:1403] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 20:04:00.843950 18508 raft_consensus.cc:359] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.844234 18508 raft_consensus.cc:385] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:00.844347 18508 raft_consensus.cc:740] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:00.844614 18508 consensus_queue.cc:260] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.845021 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.845232 18508 ts_tablet_manager.cc:1434] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:00.845322 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.845561 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.845644 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.849730 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.003s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.850986 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.851547 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.851931 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.852341 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.852699 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.853132 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.855826 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":209,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.855978 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.004s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":729,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":164,"lbm_read_time_us":149,"lbm_reads_lt_1ms":8,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"mutex_wait_us":3817,"reinsert_count":0,"update_count":5}
I20260812 20:04:00.861428 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.003s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":729,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":99,"lbm_read_time_us":150,"lbm_reads_lt_1ms":8,"lbm_write_time_us":159,"lbm_writes_lt_1ms":4,"mutex_wait_us":2881,"reinsert_count":0,"update_count":5}
I20260812 20:04:00.861876 18620 raft_consensus.cc:493] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.862123 18620 raft_consensus.cc:515] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.862910 18620 leader_election.cc:290] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.863586 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.863590 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.863845 18127 raft_consensus.cc:2468] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.863897 17964 raft_consensus.cc:2468] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:00.864408 18208 leader_election.cc:304] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.864748 18620 raft_consensus.cc:2804] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.864884 18620 raft_consensus.cc:493] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.865010 18620 raft_consensus.cc:3060] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.867465 18620 raft_consensus.cc:515] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.868170 18620 leader_election.cc:290] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.868811 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.868897 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f53efb0b87354de88e17f80c56b9eec5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:00.869099 18127 raft_consensus.cc:3060] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.869100 17964 raft_consensus.cc:3060] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.872367 18127 raft_consensus.cc:2468] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.872350 17964 raft_consensus.cc:2468] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:00.873127 18207 leader_election.cc:304] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.873523 18620 raft_consensus.cc:2804] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.873770 18620 raft_consensus.cc:697] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:00.874109 18620 consensus_queue.cc:237] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.876143 18406 raft_consensus.cc:493] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.876340 18406 raft_consensus.cc:515] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.877131 18406 leader_election.cc:290] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.878068 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.877851 17769 catalog_manager.cc:5719] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:00.878320 18279 raft_consensus.cc:2468] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:00.878808 18043 leader_election.cc:304] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:00.878782 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.879017 17964 raft_consensus.cc:2468] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:00.879164 18406 raft_consensus.cc:2804] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:00.879280 18406 raft_consensus.cc:493] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:00.879400 18406 raft_consensus.cc:3060] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.880604 18733 raft_consensus.cc:493] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.880802 18733 raft_consensus.cc:515] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.881568 18733 leader_election.cc:290] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.882335 18406 raft_consensus.cc:515] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.882807 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.883054 18406 leader_election.cc:290] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.883985 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:00.884225 17961 raft_consensus.cc:3060] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.884195 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006049a466af4197a4f71989bf9e1b50" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.884232 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa6bab2829f436ea657c5e78a693d16" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:00.884452 17964 raft_consensus.cc:2393] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d222b63b24184ca6bc530a1547fe3297 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:00.884430 18279 raft_consensus.cc:3060] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.884967 18043 leader_election.cc:304] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.885799 18406 raft_consensus.cc:3060] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.887458 18279 raft_consensus.cc:2468] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.887480 17961 raft_consensus.cc:2468] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:00.888063 18043 leader_election.cc:304] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:00.888432 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.888732 18733 raft_consensus.cc:2804] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:00.888821 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.888936 18733 raft_consensus.cc:697] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:00.889159 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.889263 18733 consensus_queue.cc:237] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.889717 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.890103 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.890479 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.891237 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.891688 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.893419 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.893857 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.894217 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.894333 18713 raft_consensus.cc:493] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.894557 18713 raft_consensus.cc:515] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.894691 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.895079 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.895401 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.895429 18713 leader_election.cc:290] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.895396 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.896140 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.896365 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.896570 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2328bed2bc24cf29b316636d7f493ba" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.896981 18127 raft_consensus.cc:2393] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:00.897518 17884 leader_election.cc:304] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.898207 18713 raft_consensus.cc:3060] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.900527 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.902462 18713 raft_consensus.cc:2749] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.906839 18406 raft_consensus.cc:2749] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.908691 17768 catalog_manager.cc:5719] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:00.909250 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.910557 18733 consensus_queue.cc:1048] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.914414 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.003s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":769,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":4976,"lbm_read_time_us":131,"lbm_reads_lt_1ms":8,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"mutex_wait_us":6536,"reinsert_count":0,"update_count":6}
I20260812 20:04:00.916651 18127 raft_consensus.cc:3060] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.921594 18406 consensus_queue.cc:1048] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.921790 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.922161 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.922475 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.925607 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.926038 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.926321 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.926608 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:00.926878 18713 raft_consensus.cc:493] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.926928 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.927073 18713 raft_consensus.cc:515] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.927276 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.927516 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.927860 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.927881 18713 leader_election.cc:290] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:00.932165 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.932466 18126 raft_consensus.cc:2393] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:00.934981 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":12011}
I20260812 20:04:00.934983 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":11236}
I20260812 20:04:00.935671 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":244,"lbm_writes_lt_1ms":4,"mutex_wait_us":6446,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.936264 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.009s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":769,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2277,"lbm_read_time_us":196,"lbm_reads_lt_1ms":8,"lbm_write_time_us":197,"lbm_writes_lt_1ms":4,"mutex_wait_us":19863,"reinsert_count":0,"update_count":6}
I20260812 20:04:00.936878 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.937156 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.937399 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:00.937836 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.938158 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:00.938393 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":201,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:00.946352 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":795,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":236,"lbm_read_time_us":193,"lbm_reads_lt_1ms":12,"lbm_write_time_us":169,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":7}
I20260812 20:04:00.947405 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0229dd61d8b419ca2c4ecbf1940d6d2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:00.948741 17883 leader_election.cc:304] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:00.949105 18591 raft_consensus.cc:3060] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.950598 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.006s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":795,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":58,"lbm_read_time_us":173,"lbm_reads_lt_1ms":8,"lbm_write_time_us":185,"lbm_writes_lt_1ms":4,"mutex_wait_us":10225,"reinsert_count":0,"update_count":7}
I20260812 20:04:00.951431 18620 raft_consensus.cc:493] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:00.951668 18620 raft_consensus.cc:515] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:00.952280 18591 raft_consensus.cc:2749] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.952502 18620 leader_election.cc:290] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:00.953685 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:00.954056 18127 raft_consensus.cc:2393] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2aa0359db6f441889edca36cd80c4058 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:04:00.955610 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae54ad4368314d4cb8713bfb2ef2e8b5" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:00.956516 18207 leader_election.cc:304] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:00.957206 18620 raft_consensus.cc:3060] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:00.958222 18717 consensus_queue.cc:1048] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:00.960503 18620 raft_consensus.cc:2749] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:00.974399 18620 consensus_queue.cc:1048] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.025830 17769 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-2" } modify_external_catalogs: true
I20260812 20:04:01.026333 17769 catalog_manager.cc:2781] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-2" } modify_external_catalogs: true
I20260812 20:04:01.040459 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.040908 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.041318 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 3bd14ad61b704e56a9dc1f37ce97925c
I20260812 20:04:01.042536 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b4a4e1ea1f52461c8aec59967842b390
I20260812 20:04:01.042850 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.043764 18765 tablet_replica.cc:333] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.044152 18765 raft_consensus.cc:2243] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.044377 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.044382 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bd14ad61b704e56a9dc1f37ce97925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.044530 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet b4a4e1ea1f52461c8aec59967842b390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.044611 18765 raft_consensus.cc:2272] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.042853 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.045140 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.045385 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.431094
I20260812 20:04:01.045497 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.045807 18765 ts_tablet_manager.cc:1916] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.045809 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.045804 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.431094
I20260812 20:04:01.046094 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.046164 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.431094
I20260812 20:04:01.046573 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.046679 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.047276 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.047695 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.048600 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet b4a4e1ea1f52461c8aec59967842b390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.048985 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.049264 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.049460 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.052433 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.431094
I20260812 20:04:01.052858 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.053063 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.053519 18765 ts_tablet_manager.cc:1929] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.053685 18765 log.cc:1199] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/3bd14ad61b704e56a9dc1f37ce97925c
I20260812 20:04:01.054212 18765 ts_tablet_manager.cc:1950] T 3bd14ad61b704e56a9dc1f37ce97925c P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.056393 18765 tablet_replica.cc:333] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.056674 18765 raft_consensus.cc:2243] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.057093 18765 raft_consensus.cc:2272] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.058284 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 3bd14ad61b704e56a9dc1f37ce97925c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.058440 18765 ts_tablet_manager.cc:1916] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.060988 18713 consensus_queue.cc:1048] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.061728 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.006s sys 0.000s Metrics: {"bytes_written":928,"delete_count":0,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:04:01.062392 18767 tablet_replica.cc:333] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.063659 18767 raft_consensus.cc:2243] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.064440 18767 raft_consensus.cc:2272] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.065464 17964 raft_consensus.cc:3060] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.066039 18765 ts_tablet_manager.cc:1929] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.066202 18765 log.cc:1199] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b4a4e1ea1f52461c8aec59967842b390
I20260812 20:04:01.066740 18765 ts_tablet_manager.cc:1950] T b4a4e1ea1f52461c8aec59967842b390 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.066766 18767 ts_tablet_manager.cc:1916] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.067569 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 18b5f38beca5449ebd3f8bcb0f0284d6
I20260812 20:04:01.067901 18766 tablet_replica.cc:333] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.067939 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b4a4e1ea1f52461c8aec59967842b390 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.068114 18766 raft_consensus.cc:2243] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.068353 18766 raft_consensus.cc:2272] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.068451 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.069417 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet b4a4e1ea1f52461c8aec59967842b390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.069473 18733 consensus_queue.cc:1048] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.070588 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 06b17a7a289a449397777c720ca2af7b
I20260812 20:04:01.070842 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.071245 18766 ts_tablet_manager.cc:1916] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.071653 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 06b17a7a289a449397777c720ca2af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.072351 18765 tablet_replica.cc:333] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.072572 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 06b17a7a289a449397777c720ca2af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.072610 18765 raft_consensus.cc:2243] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.072809 18765 raft_consensus.cc:2272] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.074680 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet ae54ad4368314d4cb8713bfb2ef2e8b5
I20260812 20:04:01.075012 18765 ts_tablet_manager.cc:1916] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.075593 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.076352 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.072567 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet 06b17a7a289a449397777c720ca2af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.077127 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet efa6bab2829f436ea657c5e78a693d16
I20260812 20:04:01.076720 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet ae54ad4368314d4cb8713bfb2ef2e8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.076879 18724 consensus_queue.cc:1048] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.078130 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.006s sys 0.000s Metrics: {"bytes_written":928,"delete_count":0,"lbm_write_time_us":197,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:04:01.078312 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet efa6bab2829f436ea657c5e78a693d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.076972 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet 18b5f38beca5449ebd3f8bcb0f0284d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.080263 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet efa6bab2829f436ea657c5e78a693d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.080266 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet efa6bab2829f436ea657c5e78a693d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.084560 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":18477,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1587,"lbm_read_time_us":673,"lbm_reads_lt_1ms":27,"lbm_write_time_us":1607,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":8,"mutex_wait_us":2144,"peak_mem_usage":32768,"reinsert_count":0,"update_count":24}
I20260812 20:04:01.088862 18724 consensus_queue.cc:1048] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.080487 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet c17c36cd2a6648c49342725074cdc96e
I20260812 20:04:01.090220 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet c17c36cd2a6648c49342725074cdc96e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.092088 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet c17c36cd2a6648c49342725074cdc96e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.092806 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet c17c36cd2a6648c49342725074cdc96e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.098052 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet cb5c495f458246c08c470376ab0dae29
I20260812 20:04:01.098951 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet cb5c495f458246c08c470376ab0dae29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.103252 18767 ts_tablet_manager.cc:1929] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.103467 18767 log.cc:1199] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/3bd14ad61b704e56a9dc1f37ce97925c
I20260812 20:04:01.106015 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet cb5c495f458246c08c470376ab0dae29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.107132 18765 ts_tablet_manager.cc:1929] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.107445 18767 ts_tablet_manager.cc:1950] T 3bd14ad61b704e56a9dc1f37ce97925c P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.107865 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet f53efb0b87354de88e17f80c56b9eec5
I20260812 20:04:01.109860 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet f53efb0b87354de88e17f80c56b9eec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.111187 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet f0229dd61d8b419ca2c4ecbf1940d6d2
I20260812 20:04:01.110203 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.113268 18767 tablet_replica.cc:333] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.113547 18767 raft_consensus.cc:2243] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.113881 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 3bd14ad61b704e56a9dc1f37ce97925c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.113952 18765 log.cc:1199] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/18b5f38beca5449ebd3f8bcb0f0284d6
I20260812 20:04:01.108835 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet f53efb0b87354de88e17f80c56b9eec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.114519 18767 raft_consensus.cc:2272] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.114985 18765 ts_tablet_manager.cc:1950] T 18b5f38beca5449ebd3f8bcb0f0284d6 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.115378 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.116391 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet cb5c495f458246c08c470376ab0dae29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.116392 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.070s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":18477,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":166,"lbm_read_time_us":5420,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":26,"lbm_write_time_us":528,"lbm_writes_lt_1ms":9,"mutex_wait_us":7161,"peak_mem_usage":32768,"reinsert_count":0,"update_count":24}
I20260812 20:04:01.117350 18244 tablet_service.cc:1558] Processing DeleteTablet for tablet 006049a466af4197a4f71989bf9e1b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.117980 18766 ts_tablet_manager.cc:1929] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.118091 18767 ts_tablet_manager.cc:1916] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.118007 18481 consensus_queue.cc:1048] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.118551 18249 tablet_service.cc:1558] Processing DeleteTablet for tablet f53efb0b87354de88e17f80c56b9eec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.118870 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet 5aac4ecab28843558e26507a083afd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.119105 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet b2328bed2bc24cf29b316636d7f493ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.120102 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet f53efb0b87354de88e17f80c56b9eec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.122735 18471 consensus_queue.cc:1048] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.125228 18767 ts_tablet_manager.cc:1929] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.125433 18767 log.cc:1199] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b4a4e1ea1f52461c8aec59967842b390
I20260812 20:04:01.126006 18767 ts_tablet_manager.cc:1950] T b4a4e1ea1f52461c8aec59967842b390 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.126003 18766 log.cc:1199] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/3bd14ad61b704e56a9dc1f37ce97925c
I20260812 20:04:01.127287 18766 ts_tablet_manager.cc:1950] T 3bd14ad61b704e56a9dc1f37ce97925c P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.127485 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b4a4e1ea1f52461c8aec59967842b390 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.127568 18767 tablet_replica.cc:333] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.127831 18767 raft_consensus.cc:2243] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.128268 18767 raft_consensus.cc:2272] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.128540 18766 tablet_replica.cc:333] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.128789 18766 raft_consensus.cc:2243] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.129029 18766 raft_consensus.cc:2272] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.130105 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 3bd14ad61b704e56a9dc1f37ce97925c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.130638 18767 ts_tablet_manager.cc:1916] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.132922 18766 ts_tablet_manager.cc:1916] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.134963 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.139451 18767 ts_tablet_manager.cc:1929] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.139714 18767 log.cc:1199] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/18b5f38beca5449ebd3f8bcb0f0284d6
I20260812 20:04:01.140177 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.140360 18767 ts_tablet_manager.cc:1950] T 18b5f38beca5449ebd3f8bcb0f0284d6 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.140663 18765 tablet_replica.cc:333] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.140985 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.140961 18765 raft_consensus.cc:2243] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.141336 18765 raft_consensus.cc:2272] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.141646 18767 tablet_replica.cc:333] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.141949 18767 raft_consensus.cc:2243] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.142153 18767 raft_consensus.cc:2272] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.142480 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.144953 18767 ts_tablet_manager.cc:1916] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.145298 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.145882 18766 ts_tablet_manager.cc:1929] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.146124 18766 log.cc:1199] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b4a4e1ea1f52461c8aec59967842b390
I20260812 20:04:01.146631 18765 ts_tablet_manager.cc:1916] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.146672 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.147053 18766 ts_tablet_manager.cc:1950] T b4a4e1ea1f52461c8aec59967842b390 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.147863 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.148316 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b4a4e1ea1f52461c8aec59967842b390 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.148800 18766 tablet_replica.cc:333] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.149096 18766 raft_consensus.cc:2243] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.149580 18766 raft_consensus.cc:2272] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.148327 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5aac4ecab28843558e26507a083afd4e
I20260812 20:04:01.153050 18767 ts_tablet_manager.cc:1929] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.153149 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 5aac4ecab28843558e26507a083afd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.153290 18767 log.cc:1199] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/06b17a7a289a449397777c720ca2af7b
I20260812 20:04:01.153632 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b2328bed2bc24cf29b316636d7f493ba
I20260812 20:04:01.153764 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 5aac4ecab28843558e26507a083afd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.153865 18767 ts_tablet_manager.cc:1950] T 06b17a7a289a449397777c720ca2af7b P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.154748 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 5aac4ecab28843558e26507a083afd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.154995 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet b2328bed2bc24cf29b316636d7f493ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.155282 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet b2328bed2bc24cf29b316636d7f493ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.155318 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet b2328bed2bc24cf29b316636d7f493ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.155491 18767 tablet_replica.cc:333] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.155566 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 8b30713b37a9473395eb907e16e77c7c
I20260812 20:04:01.155843 18767 raft_consensus.cc:2243] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.156006 18765 ts_tablet_manager.cc:1929] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.156116 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet d8750126e2be4861b6a61b34af927f54
I20260812 20:04:01.156258 18765 log.cc:1199] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/06b17a7a289a449397777c720ca2af7b
I20260812 20:04:01.156311 18767 raft_consensus.cc:2272] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.156637 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 006049a466af4197a4f71989bf9e1b50
I20260812 20:04:01.156843 18765 ts_tablet_manager.cc:1950] T 06b17a7a289a449397777c720ca2af7b P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.157069 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 8b30713b37a9473395eb907e16e77c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.157168 17769 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.157599 18767 ts_tablet_manager.cc:1916] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.157876 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet 006049a466af4197a4f71989bf9e1b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.158030 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet d8750126e2be4861b6a61b34af927f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.158628 18765 tablet_replica.cc:333] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.158828 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 06b17a7a289a449397777c720ca2af7b (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.158882 18765 raft_consensus.cc:2243] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.159192 18765 raft_consensus.cc:2272] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.159155 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet d8750126e2be4861b6a61b34af927f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.159484 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet 006049a466af4197a4f71989bf9e1b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.159519 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet d8750126e2be4861b6a61b34af927f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.159740 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.159765 17927 tablet_service.cc:1558] Processing DeleteTablet for tablet 006049a466af4197a4f71989bf9e1b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.160024 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 8b30713b37a9473395eb907e16e77c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:01.160113 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 8b30713b37a9473395eb907e16e77c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:01.160384 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 06b17a7a289a449397777c720ca2af7b (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.160742 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:01.161269 17767 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38702:
name: "table-3"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260812 20:04:01.161860 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a1404f2da764dfd8797dfaad98e954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:01 UTC) from {username='slave'} at 127.0.0.1:44302
W20260812 20:04:01.162636 17767 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 20:04:01.163710 18765 ts_tablet_manager.cc:1916] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.166491 18767 ts_tablet_manager.cc:1929] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.166692 18767 log.cc:1199] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/ae54ad4368314d4cb8713bfb2ef2e8b5
I20260812 20:04:01.167286 18767 ts_tablet_manager.cc:1950] T ae54ad4368314d4cb8713bfb2ef2e8b5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.168251 18766 ts_tablet_manager.cc:1916] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.168733 18767 tablet_replica.cc:333] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.169059 18767 raft_consensus.cc:2243] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.169154 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.169677 18767 raft_consensus.cc:2272] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.170445 18767 ts_tablet_manager.cc:1916] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.171522 18765 ts_tablet_manager.cc:1929] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.171757 18765 log.cc:1199] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/ae54ad4368314d4cb8713bfb2ef2e8b5
I20260812 20:04:01.172389 18765 ts_tablet_manager.cc:1950] T ae54ad4368314d4cb8713bfb2ef2e8b5 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.173818 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.174360 18765 tablet_replica.cc:333] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.174664 18765 raft_consensus.cc:2243] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.175001 18765 pending_rounds.cc:70] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Trying to abort 1 pending ops.
I20260812 20:04:01.175168 18765 pending_rounds.cc:77] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770407485755392 op_type: NO_OP noop_request { }
I20260812 20:04:01.175354 18766 ts_tablet_manager.cc:1929] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.175393 18765 raft_consensus.cc:2889] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:01.175509 18766 log.cc:1199] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/06b17a7a289a449397777c720ca2af7b
I20260812 20:04:01.175707 18765 raft_consensus.cc:2272] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.175997 18766 ts_tablet_manager.cc:1950] T 06b17a7a289a449397777c720ca2af7b P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.177394 18767 ts_tablet_manager.cc:1929] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.177479 18766 tablet_replica.cc:333] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.177505 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 06b17a7a289a449397777c720ca2af7b (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.177701 18767 log.cc:1199] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/efa6bab2829f436ea657c5e78a693d16
I20260812 20:04:01.177840 18766 raft_consensus.cc:2243] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.178087 18766 raft_consensus.cc:2272] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.178225 18767 ts_tablet_manager.cc:1950] T efa6bab2829f436ea657c5e78a693d16 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.179477 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.179944 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.186689 18765 ts_tablet_manager.cc:1916] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.188593 18766 ts_tablet_manager.cc:1916] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.191071 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet efa6bab2829f436ea657c5e78a693d16 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.191536 18767 tablet_replica.cc:333] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.192332 18767 raft_consensus.cc:2243] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.192719 18767 pending_rounds.cc:70] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Trying to abort 1 pending ops.
I20260812 20:04:01.193702 18767 pending_rounds.cc:77] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770407082520576 op_type: NO_OP noop_request { }
I20260812 20:04:01.193964 18767 raft_consensus.cc:2889] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:01.194146 18767 raft_consensus.cc:2272] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.194589 18765 ts_tablet_manager.cc:1929] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.195286 18767 ts_tablet_manager.cc:1916] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.197319 18766 ts_tablet_manager.cc:1929] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.197520 18766 log.cc:1199] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/ae54ad4368314d4cb8713bfb2ef2e8b5
I20260812 20:04:01.198076 18766 ts_tablet_manager.cc:1950] T ae54ad4368314d4cb8713bfb2ef2e8b5 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.199376 18766 tablet_replica.cc:333] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.199601 18766 raft_consensus.cc:2243] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.199823 18766 raft_consensus.cc:2272] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.200275 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet ae54ad4368314d4cb8713bfb2ef2e8b5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.200536 18765 log.cc:1199] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/efa6bab2829f436ea657c5e78a693d16
I20260812 20:04:01.201222 18765 ts_tablet_manager.cc:1950] T efa6bab2829f436ea657c5e78a693d16 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.202731 18766 ts_tablet_manager.cc:1916] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.202986 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet efa6bab2829f436ea657c5e78a693d16 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.203472 18765 tablet_replica.cc:333] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.203783 18765 raft_consensus.cc:2243] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.204113 18765 raft_consensus.cc:2272] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.204929 18765 ts_tablet_manager.cc:1916] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.205359 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.010s sys 0.003s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1044,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260812 20:04:01.206274 18105 tablet_service.cc:1511] Processing CreateTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.206802 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e88eefade65344a4ab4ac5ac3bb9d4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.209373 18767 ts_tablet_manager.cc:1929] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.209738 18766 ts_tablet_manager.cc:1929] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.209722 18767 log.cc:1199] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/c17c36cd2a6648c49342725074cdc96e
I20260812 20:04:01.209873 18766 log.cc:1199] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/18b5f38beca5449ebd3f8bcb0f0284d6
I20260812 20:04:01.211269 17920 tablet_service.cc:1511] Processing CreateTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.211500 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 73026c3f231b469887aa3cfadf305ade (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.211283 18247 tablet_service.cc:1511] Processing CreateTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.211791 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e88eefade65344a4ab4ac5ac3bb9d4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.211977 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e88eefade65344a4ab4ac5ac3bb9d4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.212006 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73026c3f231b469887aa3cfadf305ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.214037 18100 tablet_service.cc:1511] Processing CreateTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.214440 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.216620 18515 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.216835 18242 tablet_service.cc:1511] Processing CreateTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.217301 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.218168 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 73026c3f231b469887aa3cfadf305ade (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.218544 18515 tablet_bootstrap.cc:654] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.218623 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73026c3f231b469887aa3cfadf305ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.219722 18508 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.221192 18508 tablet_bootstrap.cc:654] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.222084 18094 tablet_service.cc:1511] Processing CreateTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.222084 18257 tablet_service.cc:1511] Processing CreateTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.222524 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50886c57ca84a5ea6c5e5b1278de7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.222558 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50886c57ca84a5ea6c5e5b1278de7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.223412 18249 tablet_service.cc:1511] Processing CreateTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.223912 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5128231e073f4f5c8bd9697a9c23a82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.226034 18104 tablet_service.cc:1511] Processing CreateTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.226450 18104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5128231e073f4f5c8bd9697a9c23a82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.211582 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 73026c3f231b469887aa3cfadf305ade (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.226881 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73026c3f231b469887aa3cfadf305ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.227998 18724 consensus_queue.cc:1048] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.230103 17927 tablet_service.cc:1511] Processing CreateTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.230590 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.232555 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.232995 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9590af0a4c9543bfb8f07837c115ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.233875 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.234002 18091 tablet_service.cc:1511] Processing CreateTablet for tablet f922ded62e854f609412e537f21eb9b1 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.234257 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:01.234464 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f922ded62e854f609412e537f21eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.234584 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.234891 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.236655 18657 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.237355 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20260812 20:04:01.238030 18098 tablet_service.cc:1511] Processing CreateTablet for tablet de0311de0f984639b73eddfef7076109 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.238512 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0311de0f984639b73eddfef7076109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.238688 18657 tablet_bootstrap.cc:654] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.241920 18087 tablet_service.cc:1511] Processing CreateTablet for tablet af421db565214a209958d4ab416c71e3 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.242378 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af421db565214a209958d4ab416c71e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.242599 17921 tablet_service.cc:1511] Processing CreateTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.210393 18766 ts_tablet_manager.cc:1950] T 18b5f38beca5449ebd3f8bcb0f0284d6 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.243088 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5128231e073f4f5c8bd9697a9c23a82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.243458 18620 consensus_queue.cc:1048] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.244165 18766 tablet_replica.cc:333] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.244375 18766 raft_consensus.cc:2243] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.244563 18766 raft_consensus.cc:2272] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.245260 18766 ts_tablet_manager.cc:1916] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.246031 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.245999 17920 tablet_service.cc:1511] Processing CreateTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.246393 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.246439 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e19899d4140ac8f708e183f2cd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.246532 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50886c57ca84a5ea6c5e5b1278de7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.249174 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 5743e1151ef0487ab67f6478b6126279 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.249660 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5743e1151ef0487ab67f6478b6126279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.249922 18620 consensus_queue.cc:1048] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.254920 18766 ts_tablet_manager.cc:1929] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.255098 18766 log.cc:1199] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/efa6bab2829f436ea657c5e78a693d16
I20260812 20:04:01.255551 18766 ts_tablet_manager.cc:1950] T efa6bab2829f436ea657c5e78a693d16 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.256887 18766 tablet_replica.cc:333] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.257112 18766 raft_consensus.cc:2243] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.257306 18766 raft_consensus.cc:2272] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.259193 18515 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.259395 18515 ts_tablet_manager.cc:1403] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20260812 20:04:01.259454 18657 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.259609 18657 ts_tablet_manager.cc:1403] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20260812 20:04:01.260351 18515 raft_consensus.cc:359] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.260442 18657 raft_consensus.cc:359] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.260572 18515 raft_consensus.cc:385] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.260630 18657 raft_consensus.cc:385] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.260669 18515 raft_consensus.cc:740] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.260743 18657 raft_consensus.cc:740] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.260931 18515 consensus_queue.cc:260] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.261031 18657 consensus_queue.cc:260] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.261528 18515 ts_tablet_manager.cc:1434] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 20:04:01.261749 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":18708}
I20260812 20:04:01.262195 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.262889 18657 ts_tablet_manager.cc:1434] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 20:04:01.263181 18657 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.267385 18657 tablet_bootstrap.cc:654] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.268513 18766 ts_tablet_manager.cc:1916] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.268532 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.561465
I20260812 20:04:01.268896 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.561465
I20260812 20:04:01.275580 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.276367 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.424819
I20260812 20:04:01.278848 18106 tablet_service.cc:1511] Processing CreateTablet for tablet b6d9104457094da480d79c4e41fe9f77 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.279371 18106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d9104457094da480d79c4e41fe9f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.281869 18766 ts_tablet_manager.cc:1929] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.282043 18766 log.cc:1199] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/c17c36cd2a6648c49342725074cdc96e
I20260812 20:04:01.286072 18724 consensus_queue.cc:1048] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.290946 18766 ts_tablet_manager.cc:1950] T c17c36cd2a6648c49342725074cdc96e P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.293723 18766 tablet_replica.cc:333] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.294209 18515 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.294575 18766 raft_consensus.cc:2243] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.295168 18766 pending_rounds.cc:70] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Trying to abort 1 pending ops.
I20260812 20:04:01.295581 18766 pending_rounds.cc:77] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770406867996672 op_type: NO_OP noop_request { }
I20260812 20:04:01.295929 18766 raft_consensus.cc:2889] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:01.296272 18766 raft_consensus.cc:2272] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.296535 18515 tablet_bootstrap.cc:654] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.297555 18766 ts_tablet_manager.cc:1916] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.303330 18257 tablet_service.cc:1511] Processing CreateTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.303330 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.123s user 0.009s sys 0.004s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":910,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":621,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":898,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260812 20:04:01.303905 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9590af0a4c9543bfb8f07837c115ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.304441 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.304849 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:01.305163 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:01.305430 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.306252 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20260812 20:04:01.306739 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.307061 18247 tablet_service.cc:1511] Processing CreateTablet for tablet f922ded62e854f609412e537f21eb9b1 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.307646 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f922ded62e854f609412e537f21eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.308498 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.005s sys 0.001s Metrics: {"bytes_written":622,"delete_count":0,"lbm_write_time_us":168,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":11}
I20260812 20:04:01.309300 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.310037 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.072s user 0.005s sys 0.001s Metrics: {"bytes_written":497,"delete_count":0,"lbm_write_time_us":204,"lbm_writes_lt_1ms":4,"mutex_wait_us":16792,"reinsert_count":0,"update_count":5}
I20260812 20:04:01.310724 18242 tablet_service.cc:1511] Processing CreateTablet for tablet de0311de0f984639b73eddfef7076109 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.311264 17922 tablet_service.cc:1511] Processing CreateTablet for tablet f922ded62e854f609412e537f21eb9b1 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.311533 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0311de0f984639b73eddfef7076109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.311802 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f922ded62e854f609412e537f21eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.313160 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.002s Metrics: {"bytes_written":707,"delete_count":0,"lbm_write_time_us":308,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260812 20:04:01.314306 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.354742
I20260812 20:04:01.314306 17938 tablet_service.cc:1511] Processing CreateTablet for tablet de0311de0f984639b73eddfef7076109 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.314759 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0311de0f984639b73eddfef7076109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.314800 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.354742
I20260812 20:04:01.317893 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.272354
I20260812 20:04:01.318235 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.326845 17921 tablet_service.cc:1511] Processing CreateTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.326984 18767 ts_tablet_manager.cc:1950] T c17c36cd2a6648c49342725074cdc96e P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.327435 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9590af0a4c9543bfb8f07837c115ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.328169 18767 tablet_replica.cc:333] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.328433 18767 raft_consensus.cc:2243] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.328693 18767 raft_consensus.cc:2272] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.335399 18717 raft_consensus.cc:493] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.335587 18717 raft_consensus.cc:515] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.336431 18717 leader_election.cc:290] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:01.336622 18717 consensus_queue.cc:1048] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.339807 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:01.340695 18208 leader_election.cc:343] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.340973 18724 consensus_queue.cc:1048] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.342487 18508 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.342658 18767 ts_tablet_manager.cc:1916] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.343982 18766 ts_tablet_manager.cc:1929] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.344149 18766 log.cc:1199] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/f0229dd61d8b419ca2c4ecbf1940d6d2
I20260812 20:04:01.344693 18766 ts_tablet_manager.cc:1950] T f0229dd61d8b419ca2c4ecbf1940d6d2 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.342667 18508 ts_tablet_manager.cc:1403] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20260812 20:04:01.347252 18508 raft_consensus.cc:359] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.347488 18508 raft_consensus.cc:385] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.347391 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:01.347604 18508 raft_consensus.cc:740] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.348067 18508 consensus_queue.cc:260] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.348819 18508 ts_tablet_manager.cc:1434] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 20:04:01.349170 18508 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
W20260812 20:04:01.350390 18207 leader_election.cc:343] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.350642 18207 leader_election.cc:304] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:01.351159 18717 raft_consensus.cc:2749] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:01.351228 18508 tablet_bootstrap.cc:654] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.355347 18242 tablet_service.cc:1511] Processing CreateTablet for tablet af421db565214a209958d4ab416c71e3 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.355500 17937 tablet_service.cc:1511] Processing CreateTablet for tablet af421db565214a209958d4ab416c71e3 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.355875 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af421db565214a209958d4ab416c71e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.355875 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af421db565214a209958d4ab416c71e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.358024 17927 tablet_service.cc:1511] Processing CreateTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.358615 18247 tablet_service.cc:1511] Processing CreateTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.359068 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e19899d4140ac8f708e183f2cd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.362039 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 5743e1151ef0487ab67f6478b6126279 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.362500 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5743e1151ef0487ab67f6478b6126279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.367436 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet c17c36cd2a6648c49342725074cdc96e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.367985 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e19899d4140ac8f708e183f2cd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.368356 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 18b5f38beca5449ebd3f8bcb0f0284d6 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.368456 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.368582 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet efa6bab2829f436ea657c5e78a693d16 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.380431 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.381043 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8b11ecadeb4e25b4eb9abc97920d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.383183 17922 tablet_service.cc:1511] Processing CreateTablet for tablet 5743e1151ef0487ab67f6478b6126279 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.383841 18249 tablet_service.cc:1511] Processing CreateTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.384413 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.115s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":1998,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":19277,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":539,"lbm_read_time_us":525,"lbm_reads_lt_1ms":27,"lbm_write_time_us":430,"lbm_writes_lt_1ms":9,"mutex_wait_us":19041,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260812 20:04:01.385715 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8b11ecadeb4e25b4eb9abc97920d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.386106 17756 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.387652 17938 tablet_service.cc:1511] Processing CreateTablet for tablet b6d9104457094da480d79c4e41fe9f77 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.388118 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8b11ecadeb4e25b4eb9abc97920d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.402766 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.088s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1687,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":264,"lbm_read_time_us":538,"lbm_reads_lt_1ms":21,"lbm_write_time_us":392,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260812 20:04:01.401010 18765 ts_tablet_manager.cc:1929] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.404948 18765 log.cc:1199] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/c17c36cd2a6648c49342725074cdc96e
I20260812 20:04:01.387279 18657 tablet_bootstrap.cc:492] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.389766 18766 tablet_replica.cc:333] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.406272 18657 ts_tablet_manager.cc:1403] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.143s user 0.004s sys 0.017s
I20260812 20:04:01.406335 18766 raft_consensus.cc:2243] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.406780 18766 raft_consensus.cc:2272] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.407400 18657 raft_consensus.cc:359] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.407658 18657 raft_consensus.cc:385] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.407763 18657 raft_consensus.cc:740] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.408196 18766 ts_tablet_manager.cc:1916] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.408226 18765 ts_tablet_manager.cc:1950] T c17c36cd2a6648c49342725074cdc96e P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.409113 17667 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.409546 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet c17c36cd2a6648c49342725074cdc96e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.413669 18724 consensus_queue.cc:1048] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 20:04:01.414371 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d9104457094da480d79c4e41fe9f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.415951 18766 ts_tablet_manager.cc:1929] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.416276 18766 log.cc:1199] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/006049a466af4197a4f71989bf9e1b50
I20260812 20:04:01.416505 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet c17c36cd2a6648c49342725074cdc96e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.417768 18766 ts_tablet_manager.cc:1950] T 006049a466af4197a4f71989bf9e1b50 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.419137 18766 tablet_replica.cc:333] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.419399 18766 raft_consensus.cc:2243] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.419441 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.408083 18657 consensus_queue.cc:260] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.419821 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 006049a466af4197a4f71989bf9e1b50 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.420853 18657 ts_tablet_manager.cc:1434] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20260812 20:04:01.421115 18765 tablet_replica.cc:333] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.421265 18766 raft_consensus.cc:2272] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.421675 18657 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.422943 18766 ts_tablet_manager.cc:1916] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.422998 18257 tablet_service.cc:1511] Processing CreateTablet for tablet b6d9104457094da480d79c4e41fe9f77 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.421747 18765 raft_consensus.cc:2243] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.423578 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d9104457094da480d79c4e41fe9f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.424165 18657 tablet_bootstrap.cc:654] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.427109 18508 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.427743 18767 ts_tablet_manager.cc:1929] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.427947 18767 log.cc:1199] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/cb5c495f458246c08c470376ab0dae29
I20260812 20:04:01.428529 18767 ts_tablet_manager.cc:1950] T cb5c495f458246c08c470376ab0dae29 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.429867 18767 tablet_replica.cc:333] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.430218 18767 raft_consensus.cc:2243] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.430452 18767 raft_consensus.cc:2272] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.431303 18767 ts_tablet_manager.cc:1916] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.431504 18508 ts_tablet_manager.cc:1403] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.008s
I20260812 20:04:01.432655 18508 raft_consensus.cc:359] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.433286 18508 raft_consensus.cc:385] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.434273 18508 raft_consensus.cc:740] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.434916 18508 consensus_queue.cc:260] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.436151 18508 ts_tablet_manager.cc:1434] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 20:04:01.436365 18765 raft_consensus.cc:2272] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.436728 18508 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.436559 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 4581755097204339b4b41750e54ac482 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.437268 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5743e1151ef0487ab67f6478b6126279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.437901 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4581755097204339b4b41750e54ac482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.436800 18106 tablet_service.cc:1511] Processing CreateTablet for tablet 7576613d620548df8dbd661bb74dca45 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.442121 18106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7576613d620548df8dbd661bb74dca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.446571 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet cb5c495f458246c08c470376ab0dae29 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.437039 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.447319 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5eebe47827446aae93c30cdaa4e345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.448195 18508 tablet_bootstrap.cc:654] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.452383 18765 ts_tablet_manager.cc:1916] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.458710 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.181s user 0.017s sys 0.003s Metrics: {"dirs.queue_time_us":682,"dirs.run_cpu_time_us":986,"dirs.run_wall_time_us":8074,"lbm_write_time_us":783,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"thread_start_us":68967,"threads_started":1}
I20260812 20:04:01.459990 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.467061 18242 tablet_service.cc:1511] Processing CreateTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.467238 17921 tablet_service.cc:1511] Processing CreateTablet for tablet 4581755097204339b4b41750e54ac482 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.467613 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5eebe47827446aae93c30cdaa4e345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.467648 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4581755097204339b4b41750e54ac482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.467916 17927 tablet_service.cc:1511] Processing CreateTablet for tablet 7576613d620548df8dbd661bb74dca45 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.468464 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7576613d620548df8dbd661bb74dca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.468449 18515 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.468644 18515 ts_tablet_manager.cc:1403] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.175s user 0.003s sys 0.010s
I20260812 20:04:01.468706 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 4581755097204339b4b41750e54ac482 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.469115 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4581755097204339b4b41750e54ac482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.469898 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.470947 18515 raft_consensus.cc:359] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.471184 18515 raft_consensus.cc:385] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.471251 18515 raft_consensus.cc:740] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.471534 18515 consensus_queue.cc:260] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.474143 18515 ts_tablet_manager.cc:1434] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260812 20:04:01.475379 18515 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.480531 18508 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.480726 18508 ts_tablet_manager.cc:1403] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20260812 20:04:01.481658 18508 raft_consensus.cc:359] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.481892 18508 raft_consensus.cc:385] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.482015 18508 raft_consensus.cc:740] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.482342 18508 consensus_queue.cc:260] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.482537 18724 raft_consensus.cc:493] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.482750 18724 raft_consensus.cc:515] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.483562 18724 leader_election.cc:290] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:04:01.484513 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e88eefade65344a4ab4ac5ac3bb9d4ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.484866 18127 raft_consensus.cc:2468] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:01.485396 18208 leader_election.cc:304] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:01.485522 18247 tablet_service.cc:1511] Processing CreateTablet for tablet 7576613d620548df8dbd661bb74dca45 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.485893 18724 raft_consensus.cc:2804] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.485997 18724 raft_consensus.cc:493] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.485956 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 (DEFAULT_TABLE table=table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:01.486115 18724 raft_consensus.cc:3060] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.486370 18508 ts_tablet_manager.cc:1434] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260812 20:04:01.486469 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5eebe47827446aae93c30cdaa4e345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.487303 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e88eefade65344a4ab4ac5ac3bb9d4ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:01.487574 17961 raft_consensus.cc:2468] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:01.488823 18508 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.490011 18766 ts_tablet_manager.cc:1929] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.490171 18766 log.cc:1199] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/f53efb0b87354de88e17f80c56b9eec5
I20260812 20:04:01.489455 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.171s user 0.011s sys 0.007s Metrics: {"dirs.queue_time_us":3640,"dirs.run_cpu_time_us":1069,"dirs.run_wall_time_us":10512,"lbm_write_time_us":375,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"thread_start_us":61361,"threads_started":1}
I20260812 20:04:01.490689 18766 ts_tablet_manager.cc:1950] T f53efb0b87354de88e17f80c56b9eec5 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.490955 18508 tablet_bootstrap.cc:654] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.491236 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.491880 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.491993 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet f53efb0b87354de88e17f80c56b9eec5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.492411 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.492941 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.355052
I20260812 20:04:01.501659 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7576613d620548df8dbd661bb74dca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:01.502352 18767 ts_tablet_manager.cc:1929] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.502512 18767 log.cc:1199] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/f53efb0b87354de88e17f80c56b9eec5
I20260812 20:04:01.502866 18767 ts_tablet_manager.cc:1950] T f53efb0b87354de88e17f80c56b9eec5 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.504581 18766 tablet_replica.cc:333] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.504858 18766 raft_consensus.cc:2243] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.505144 18767 tablet_replica.cc:333] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.505327 18766 raft_consensus.cc:2272] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.505342 18767 raft_consensus.cc:2243] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.505553 18767 raft_consensus.cc:2272] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.509222 18508 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.509433 18508 ts_tablet_manager.cc:1403] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20260812 20:04:01.510268 18508 raft_consensus.cc:359] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.510500 18508 raft_consensus.cc:385] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.510596 18508 raft_consensus.cc:740] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.510910 18508 consensus_queue.cc:260] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.512051 18766 ts_tablet_manager.cc:1916] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.513577 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet f53efb0b87354de88e17f80c56b9eec5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.514796 18767 ts_tablet_manager.cc:1916] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.515075 18508 ts_tablet_manager.cc:1434] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 20:04:01.517400 18508 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.519311 18724 raft_consensus.cc:515] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.519639 18508 tablet_bootstrap.cc:654] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.520394 18724 leader_election.cc:290] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337)
I20260812 20:04:01.520902 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e88eefade65344a4ab4ac5ac3bb9d4ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.521202 18127 raft_consensus.cc:3060] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.524613 18127 raft_consensus.cc:2468] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:01.525835 18208 leader_election.cc:304] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
W20260812 20:04:01.526916 18043 consensus_peers.cc:597] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:01.527722 18724 raft_consensus.cc:2804] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.528517 18657 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.528893 18657 ts_tablet_manager.cc:1403] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.031s
I20260812 20:04:01.529125 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e88eefade65344a4ab4ac5ac3bb9d4ac" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:01.529413 17961 raft_consensus.cc:3060] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.530292 18657 raft_consensus.cc:359] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.530558 18657 raft_consensus.cc:385] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.530787 18657 raft_consensus.cc:740] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.531283 18657 consensus_queue.cc:260] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.532025 18657 ts_tablet_manager.cc:1434] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 20:04:01.532385 18657 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.528678 18724 raft_consensus.cc:697] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:01.533313 18724 consensus_queue.cc:237] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } }
I20260812 20:04:01.534682 17961 raft_consensus.cc:2468] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:01.536003 17667 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.537599 18767 ts_tablet_manager.cc:1929] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.540828 18657 tablet_bootstrap.cc:654] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.543375 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.066s user 0.010s sys 0.003s Metrics: {"dirs.queue_time_us":7387,"dirs.run_cpu_time_us":1122,"dirs.run_wall_time_us":7890,"lbm_write_time_us":366,"lbm_writes_lt_1ms":7,"num_input_rowsets":0}
I20260812 20:04:01.545361 18766 ts_tablet_manager.cc:1929] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.545543 18767 log.cc:1199] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.545569 18766 log.cc:1199] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/cb5c495f458246c08c470376ab0dae29
I20260812 20:04:01.546487 18766 ts_tablet_manager.cc:1950] T cb5c495f458246c08c470376ab0dae29 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.547358 18767 ts_tablet_manager.cc:1950] T 4a1404f2da764dfd8797dfaad98e954f P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.547885 18766 tablet_replica.cc:333] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.548182 18766 raft_consensus.cc:2243] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.548884 18766 raft_consensus.cc:2272] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.550496 18766 ts_tablet_manager.cc:1916] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.551201 18767 tablet_replica.cc:333] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.551887 18508 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.552134 18508 ts_tablet_manager.cc:1403] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.001s
I20260812 20:04:01.552129 18767 raft_consensus.cc:2243] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.552867 18767 pending_rounds.cc:70] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Trying to abort 1 pending ops.
I20260812 20:04:01.553393 18767 pending_rounds.cc:77] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770406867996672 op_type: NO_OP noop_request { }
I20260812 20:04:01.554013 18508 raft_consensus.cc:359] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.554349 18508 raft_consensus.cc:385] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.554481 18508 raft_consensus.cc:740] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.554792 18508 consensus_queue.cc:260] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.555677 18508 ts_tablet_manager.cc:1434] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 20:04:01.556125 18508 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.558706 18508 tablet_bootstrap.cc:654] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.558959 18767 raft_consensus.cc:2889] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:01.561908 18767 raft_consensus.cc:2272] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.563228 18767 ts_tablet_manager.cc:1916] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.572854 18515 tablet_bootstrap.cc:654] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.575542 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4251 bytes on disk
I20260812 20:04:01.575958 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4251 bytes on disk
I20260812 20:04:01.576309 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4251 bytes on disk
I20260812 20:04:01.577351 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20260812 20:04:01.578068 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.578565 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260812 20:04:01.580178 17750 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.25.14.3:34495, user_credentials={real_user=slave}} blocked reactor thread for 53612.3us
I20260812 20:04:01.580729 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 4a1404f2da764dfd8797dfaad98e954f (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.581101 18767 ts_tablet_manager.cc:1929] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.582188 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet cb5c495f458246c08c470376ab0dae29 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.593044 18515 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.587061 18508 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.593286 18515 ts_tablet_manager.cc:1403] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.118s user 0.001s sys 0.004s
I20260812 20:04:01.593482 18767 log.cc:1199] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/f0229dd61d8b419ca2c4ecbf1940d6d2
I20260812 20:04:01.593663 18508 ts_tablet_manager.cc:1403] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.004s
I20260812 20:04:01.595244 18767 ts_tablet_manager.cc:1950] T f0229dd61d8b419ca2c4ecbf1940d6d2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.595588 18508 raft_consensus.cc:359] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.596578 18508 raft_consensus.cc:385] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.596731 18508 raft_consensus.cc:740] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.597204 18508 consensus_queue.cc:260] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.598162 18508 ts_tablet_manager.cc:1434] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 20:04:01.599399 18620 raft_consensus.cc:493] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.599807 18508 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.599807 18620 raft_consensus.cc:515] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.600636 18620 leader_election.cc:290] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:01.602253 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73026c3f231b469887aa3cfadf305ade" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:01.603159 18208 leader_election.cc:343] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.603451 18508 tablet_bootstrap.cc:654] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.604701 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73026c3f231b469887aa3cfadf305ade" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:01.605131 17961 raft_consensus.cc:2468] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:01.606168 18207 leader_election.cc:304] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.607342 18620 raft_consensus.cc:2804] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.607539 18620 raft_consensus.cc:493] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.607915 18620 raft_consensus.cc:3060] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.609503 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.610512 18767 tablet_replica.cc:333] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.611328 18767 raft_consensus.cc:2243] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.612192 18767 raft_consensus.cc:2272] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.612862 18620 raft_consensus.cc:515] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.615175 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73026c3f231b469887aa3cfadf305ade" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:01.617491 17757 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.618242 18657 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.618098 17767 catalog_manager.cc:5719] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:01.618813 18657 ts_tablet_manager.cc:1403] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.016s
I20260812 20:04:01.620296 18657 raft_consensus.cc:359] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.620787 17961 raft_consensus.cc:3060] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.621526 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73026c3f231b469887aa3cfadf305ade" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.628149 17755 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.628281 18765 ts_tablet_manager.cc:1929] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.628557 18765 log.cc:1199] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/cb5c495f458246c08c470376ab0dae29
I20260812 20:04:01.632473 18515 raft_consensus.cc:359] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.632833 18515 raft_consensus.cc:385] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.633028 18515 raft_consensus.cc:740] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.634951 17961 raft_consensus.cc:2468] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:01.635000 18515 consensus_queue.cc:260] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.635875 18767 ts_tablet_manager.cc:1916] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:04:01.622745 18208 leader_election.cc:343] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.623080 18620 leader_election.cc:290] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:01.642246 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.642347 18515 ts_tablet_manager.cc:1434] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.049s user 0.004s sys 0.000s
I20260812 20:04:01.642943 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.643121 18207 leader_election.cc:304] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.643471 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.643603 18620 raft_consensus.cc:2804] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.646306 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.646641 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.647975 18620 raft_consensus.cc:697] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:01.648428 18620 consensus_queue.cc:237] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.650154 18657 raft_consensus.cc:385] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.651144 18657 raft_consensus.cc:740] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.651804 18766 ts_tablet_manager.cc:1929] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.651760 18657 consensus_queue.cc:260] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.652683 18657 ts_tablet_manager.cc:1434] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20260812 20:04:01.653076 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.002s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":251,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:01.653963 18657 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.659981 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.160s user 0.013s sys 0.004s Metrics: {"dirs.queue_time_us":10955,"dirs.run_cpu_time_us":1180,"dirs.run_wall_time_us":6276,"lbm_write_time_us":476,"lbm_writes_lt_1ms":7,"mutex_wait_us":303,"num_input_rowsets":0}
I20260812 20:04:01.661233 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.163s user 0.013s sys 0.007s Metrics: {"dirs.queue_time_us":2512,"dirs.run_cpu_time_us":1003,"dirs.run_wall_time_us":6752,"lbm_write_time_us":14169,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":6,"mutex_wait_us":7865,"num_input_rowsets":0,"thread_start_us":43610,"threads_started":1}
I20260812 20:04:01.661861 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.662297 18767 ts_tablet_manager.cc:1929] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.662511 18767 log.cc:1199] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5aac4ecab28843558e26507a083afd4e
I20260812 20:04:01.662577 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.663195 18767 ts_tablet_manager.cc:1950] T 5aac4ecab28843558e26507a083afd4e P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.663607 18657 tablet_bootstrap.cc:654] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.664219 18515 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.668468 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5aac4ecab28843558e26507a083afd4e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.668860 18515 tablet_bootstrap.cc:654] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.670254 18765 ts_tablet_manager.cc:1950] T cb5c495f458246c08c470376ab0dae29 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.672096 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet cb5c495f458246c08c470376ab0dae29 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.672600 18767 tablet_replica.cc:333] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.672766 18765 tablet_replica.cc:333] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.673442 18765 raft_consensus.cc:2243] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.673867 18765 raft_consensus.cc:2272] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.674584 18657 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.675271 18765 ts_tablet_manager.cc:1916] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.675756 18657 ts_tablet_manager.cc:1403] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260812 20:04:01.675940 18767 raft_consensus.cc:2243] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.677016 18767 raft_consensus.cc:2272] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.677662 18766 log.cc:1199] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5aac4ecab28843558e26507a083afd4e
I20260812 20:04:01.678507 18657 raft_consensus.cc:359] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.678893 18657 raft_consensus.cc:385] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.680450 17756 tablet.cc:2106] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:01.680867 18508 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.681464 18508 ts_tablet_manager.cc:1403] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.082s user 0.033s sys 0.024s
I20260812 20:04:01.680850 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.220s user 0.052s sys 0.012s Metrics: {"bytes_written":31652,"cfile_cache_hit":15,"cfile_cache_hit_bytes":24862,"cfile_cache_miss":37,"cfile_cache_miss_bytes":34682,"cfile_init":10,"delta_iterators_relevant":10,"dirs.queue_time_us":2088,"dirs.run_cpu_time_us":1240,"dirs.run_wall_time_us":10235,"drs_written":1,"lbm_read_time_us":6104,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":75,"lbm_write_time_us":1662,"lbm_writes_lt_1ms":28,"num_input_rowsets":2,"peak_mem_usage":139412,"rows_written":71,"thread_start_us":105208,"threads_started":2}
I20260812 20:04:01.682533 18767 ts_tablet_manager.cc:1916] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.683727 18657 raft_consensus.cc:740] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.684149 18657 consensus_queue.cc:260] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.684405 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.683303 18508 raft_consensus.cc:359] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.685027 18713 raft_consensus.cc:493] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.686111 18508 raft_consensus.cc:385] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.686255 18508 raft_consensus.cc:740] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.686169 18713 raft_consensus.cc:515] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.686581 18508 consensus_queue.cc:260] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.688757 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f48b54fa13d4c1ba18d59e9dd8dbb01" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.688952 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.121s user 0.011s sys 0.010s Metrics: {"dirs.queue_time_us":39045,"dirs.run_cpu_time_us":1380,"dirs.run_wall_time_us":7010,"lbm_write_time_us":23378,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":6,"mutex_wait_us":20176,"num_input_rowsets":0}
I20260812 20:04:01.689514 18713 leader_election.cc:290] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.689886 18657 ts_tablet_manager.cc:1434] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
W20260812 20:04:01.689857 17884 leader_election.cc:343] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.690143 18715 raft_consensus.cc:493] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.690429 18657 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.690464 18715 raft_consensus.cc:515] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.691253 18715 leader_election.cc:290] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.692167 18508 ts_tablet_manager.cc:1434] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 20:04:01.692421 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.693123 18508 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
W20260812 20:04:01.694064 17884 leader_election.cc:343] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.694314 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f48b54fa13d4c1ba18d59e9dd8dbb01" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:01.694757 18277 raft_consensus.cc:2468] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:01.695964 18766 ts_tablet_manager.cc:1950] T 5aac4ecab28843558e26507a083afd4e P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.696316 18657 tablet_bootstrap.cc:654] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.697028 18508 tablet_bootstrap.cc:654] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.697721 17883 leader_election.cc:304] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.698539 18715 raft_consensus.cc:2804] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.699781 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:01.699978 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5aac4ecab28843558e26507a083afd4e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.700086 18279 raft_consensus.cc:2468] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:01.700703 18766 tablet_replica.cc:333] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.701108 18766 raft_consensus.cc:2243] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.701603 17883 leader_election.cc:304] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.701778 18766 raft_consensus.cc:2272] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.702265 18713 raft_consensus.cc:2804] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.702522 18713 raft_consensus.cc:493] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.702658 18713 raft_consensus.cc:3060] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.703433 18766 ts_tablet_manager.cc:1916] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.704546 18715 raft_consensus.cc:493] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.704906 18715 raft_consensus.cc:3060] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.707922 18657 tablet_bootstrap.cc:492] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.708321 18657 ts_tablet_manager.cc:1403] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260812 20:04:01.709770 18657 raft_consensus.cc:359] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.710537 18657 raft_consensus.cc:385] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.710784 18657 raft_consensus.cc:740] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.711273 18657 consensus_queue.cc:260] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.712090 18657 ts_tablet_manager.cc:1434] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 20:04:01.712077 18715 raft_consensus.cc:515] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.712706 18657 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.714282 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f48b54fa13d4c1ba18d59e9dd8dbb01" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.714370 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f48b54fa13d4c1ba18d59e9dd8dbb01" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:01.714648 18713 raft_consensus.cc:515] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.714756 18279 raft_consensus.cc:3060] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:01.715184 17884 leader_election.cc:343] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.715370 18713 leader_election.cc:290] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.715605 18715 leader_election.cc:290] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.716040 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:01.716079 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.716713 18277 raft_consensus.cc:3060] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:01.716922 17884 leader_election.cc:343] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.722723 17767 catalog_manager.cc:5719] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:01.723688 18515 tablet_bootstrap.cc:492] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.723999 18515 ts_tablet_manager.cc:1403] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.007s
I20260812 20:04:01.725307 18515 raft_consensus.cc:359] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.727463 18277 raft_consensus.cc:2468] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:01.728333 18279 raft_consensus.cc:2468] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:01.730374 17883 leader_election.cc:304] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.731278 17883 leader_election.cc:304] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.731858 18713 raft_consensus.cc:2804] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.731948 18715 raft_consensus.cc:2804] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.732142 18657 tablet_bootstrap.cc:654] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.732126 18713 raft_consensus.cc:697] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:01.732627 18713 consensus_queue.cc:237] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.739665 18766 ts_tablet_manager.cc:1929] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.742395 18515 raft_consensus.cc:385] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.742610 18515 raft_consensus.cc:740] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.744184 18508 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.744578 18508 ts_tablet_manager.cc:1403] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.008s
I20260812 20:04:01.745884 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.746554 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.747071 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.747645 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.748488 18766 log.cc:1199] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b2328bed2bc24cf29b316636d7f493ba
I20260812 20:04:01.749419 18515 consensus_queue.cc:260] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.750118 18766 ts_tablet_manager.cc:1950] T b2328bed2bc24cf29b316636d7f493ba P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.751298 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.751490 18766 tablet_replica.cc:333] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.751781 18766 raft_consensus.cc:2243] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.751962 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.752147 18766 raft_consensus.cc:2272] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.752172 18515 ts_tablet_manager.cc:1434] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.028s user 0.002s sys 0.002s
I20260812 20:04:01.752403 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.752852 18515 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.754776 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.755901 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b2328bed2bc24cf29b316636d7f493ba (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
W20260812 20:04:01.756474 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet f53efb0b87354de88e17f80c56b9eec5 because the tablet was not found. No further retry: Not found: Tablet not found: f53efb0b87354de88e17f80c56b9eec5
I20260812 20:04:01.756875 18766 ts_tablet_manager.cc:1916] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.758152 18765 ts_tablet_manager.cc:1929] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.758311 18765 log.cc:1199] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/f53efb0b87354de88e17f80c56b9eec5
I20260812 20:04:01.759059 18765 ts_tablet_manager.cc:1950] T f53efb0b87354de88e17f80c56b9eec5 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.762790 18715 raft_consensus.cc:697] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:01.763765 18715 consensus_queue.cc:237] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.765900 18713 raft_consensus.cc:493] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.766294 18713 raft_consensus.cc:515] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.767426 18713 leader_election.cc:290] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.767688 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":482,"lbm_writes_lt_1ms":4,"mutex_wait_us":19759,"reinsert_count":0,"spinlock_wait_cycles":827264,"update_count":1}
I20260812 20:04:01.768441 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:01.768733 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.768899 18279 raft_consensus.cc:2468] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
W20260812 20:04:01.769682 17884 leader_election.cc:343] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.770006 17884 leader_election.cc:304] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.770901 18713 raft_consensus.cc:2804] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.771106 18713 raft_consensus.cc:493] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.771399 18713 raft_consensus.cc:3060] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.774617 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet f53efb0b87354de88e17f80c56b9eec5 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.775655 18515 tablet_bootstrap.cc:654] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.777427 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.004s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":291,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:01.778877 18767 ts_tablet_manager.cc:1929] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.779160 18767 log.cc:1199] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b2328bed2bc24cf29b316636d7f493ba
I20260812 20:04:01.779484 18766 ts_tablet_manager.cc:1929] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.779752 18766 log.cc:1199] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.780009 18767 ts_tablet_manager.cc:1950] T b2328bed2bc24cf29b316636d7f493ba P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.780330 18766 ts_tablet_manager.cc:1950] T 4a1404f2da764dfd8797dfaad98e954f P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.780258 18713 raft_consensus.cc:515] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.781311 18767 tablet_replica.cc:333] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.781497 18508 raft_consensus.cc:359] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.781937 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 4a1404f2da764dfd8797dfaad98e954f (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.782241 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b2328bed2bc24cf29b316636d7f493ba (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.781098 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.007s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":3220,"lbm_read_time_us":229,"lbm_reads_lt_1ms":8,"lbm_write_time_us":469,"lbm_writes_lt_1ms":4,"mutex_wait_us":1640,"reinsert_count":0,"update_count":2}
I20260812 20:04:01.782009 18508 raft_consensus.cc:385] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.783176 18508 raft_consensus.cc:740] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.784173 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.784312 18508 consensus_queue.cc:260] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.785310 18508 ts_tablet_manager.cc:1434] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
W20260812 20:04:01.785497 17884 leader_election.cc:343] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.786027 18508 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.786991 18765 tablet_replica.cc:333] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.787421 18765 raft_consensus.cc:2243] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:01.788182 18765 raft_consensus.cc:2272] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 20:04:01.788893 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet f0229dd61d8b419ca2c4ecbf1940d6d2 because the tablet was not found. No further retry: Not found: Tablet not found: f0229dd61d8b419ca2c4ecbf1940d6d2
W20260812 20:04:01.790015 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet 5aac4ecab28843558e26507a083afd4e because the tablet was not found. No further retry: Not found: Tablet not found: 5aac4ecab28843558e26507a083afd4e
I20260812 20:04:01.790091 18765 ts_tablet_manager.cc:1916] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.790885 18508 tablet_bootstrap.cc:654] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.791437 18767 raft_consensus.cc:2243] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
W20260812 20:04:01.791978 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet b2328bed2bc24cf29b316636d7f493ba because the tablet was not found. No further retry: Not found: Tablet not found: b2328bed2bc24cf29b316636d7f493ba
I20260812 20:04:01.792444 18766 tablet_replica.cc:333] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.792769 18766 raft_consensus.cc:2243] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.793699 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b50886c57ca84a5ea6c5e5b1278de7ac" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:01.794248 18279 raft_consensus.cc:3060] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.795459 18657 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.795725 18657 ts_tablet_manager.cc:1403] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.009s
I20260812 20:04:01.796052 17767 catalog_manager.cc:5719] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:01.796304 18767 raft_consensus.cc:2272] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.796651 17767 catalog_manager.cc:5719] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:01.794243 18713 leader_election.cc:290] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.797273 18657 raft_consensus.cc:359] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.797773 18657 raft_consensus.cc:385] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.798014 18657 raft_consensus.cc:740] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.798039 18767 ts_tablet_manager.cc:1916] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.798395 18657 consensus_queue.cc:260] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.799156 18657 ts_tablet_manager.cc:1434] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:04:01.799572 18657 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.802651 18657 tablet_bootstrap.cc:654] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.815359 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.815711 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.815972 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.816202 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.816545 18279 raft_consensus.cc:2468] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:01.817287 17883 leader_election.cc:304] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.817499 18657 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.817703 18657 ts_tablet_manager.cc:1403] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20260812 20:04:01.818071 18713 raft_consensus.cc:2804] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.818367 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.818457 18713 raft_consensus.cc:697] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:01.818706 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.818985 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.818830 18713 consensus_queue.cc:237] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.820189 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.820513 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.820834 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.820797 18766 raft_consensus.cc:2272] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.821213 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.821414 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.822306 18766 ts_tablet_manager.cc:1916] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.822579 18657 raft_consensus.cc:359] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.822881 18657 raft_consensus.cc:385] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.823002 18657 raft_consensus.cc:740] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.823423 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.823390 18657 consensus_queue.cc:260] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.824155 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.824483 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4251 bytes on disk
I20260812 20:04:01.824757 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:01.825820 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.825778 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260812 20:04:01.828498 18765 ts_tablet_manager.cc:1929] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.830991 18657 ts_tablet_manager.cc:1434] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260812 20:04:01.831547 18657 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.833423 17769 catalog_manager.cc:5719] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:01.830687 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":213,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:01.834779 18657 tablet_bootstrap.cc:654] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.842470 18765 log.cc:1199] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.842577 18515 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.842806 18515 ts_tablet_manager.cc:1403] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.000s
I20260812 20:04:01.843227 18765 ts_tablet_manager.cc:1950] T 4a1404f2da764dfd8797dfaad98e954f P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.843755 18766 ts_tablet_manager.cc:1929] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.844201 18766 log.cc:1199] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/d8750126e2be4861b6a61b34af927f54
I20260812 20:04:01.844795 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 4a1404f2da764dfd8797dfaad98e954f (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.845098 18766 ts_tablet_manager.cc:1950] T d8750126e2be4861b6a61b34af927f54 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.845232 18515 raft_consensus.cc:359] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.845453 18765 tablet_replica.cc:333] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.845530 18515 raft_consensus.cc:385] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.845800 18515 raft_consensus.cc:740] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.846617 18765 raft_consensus.cc:2243] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.846966 18765 raft_consensus.cc:2272] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.841306 18767 ts_tablet_manager.cc:1929] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.847325 18515 consensus_queue.cc:260] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.847467 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet d8750126e2be4861b6a61b34af927f54 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.847527 18767 log.cc:1199] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/d8750126e2be4861b6a61b34af927f54
I20260812 20:04:01.848207 18767 ts_tablet_manager.cc:1950] T d8750126e2be4861b6a61b34af927f54 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.849030 18515 ts_tablet_manager.cc:1434] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260812 20:04:01.849442 18515 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.853602 18515 tablet_bootstrap.cc:654] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.853955 18766 tablet_replica.cc:333] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:01.854212 18766 raft_consensus.cc:2243] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.854880 18713 raft_consensus.cc:493] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.855091 18713 raft_consensus.cc:515] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.856076 18713 leader_election.cc:290] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.856441 18765 ts_tablet_manager.cc:1916] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.857100 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.857810 18766 raft_consensus.cc:2272] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.857851 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:01.858057 18279 raft_consensus.cc:2468] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:01.858413 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.858747 17883 leader_election.cc:304] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:01.858968 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.859342 18766 ts_tablet_manager.cc:1916] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:04:01.859548 17884 leader_election.cc:343] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.860121 18713 raft_consensus.cc:2804] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.860320 18713 raft_consensus.cc:493] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 20:04:01.860308 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet 006049a466af4197a4f71989bf9e1b50 because the tablet was not found. No further retry: Not found: Tablet not found: 006049a466af4197a4f71989bf9e1b50
I20260812 20:04:01.860463 18713 raft_consensus.cc:3060] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:01.860982 17750 catalog_manager.cc:5017] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): delete failed for tablet 4a1404f2da764dfd8797dfaad98e954f because the tablet was not found. No further retry: Not found: Tablet not found: 4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.864154 18713 raft_consensus.cc:515] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.866109 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
W20260812 20:04:01.867571 17884 leader_election.cc:343] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.868600 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.869207 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.869678 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:01.869880 18766 ts_tablet_manager.cc:1929] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.869980 18279 raft_consensus.cc:3060] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.870146 18766 log.cc:1199] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/8b30713b37a9473395eb907e16e77c7c
I20260812 20:04:01.870841 18766 ts_tablet_manager.cc:1950] T 8b30713b37a9473395eb907e16e77c7c P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:01.872449 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 8b30713b37a9473395eb907e16e77c7c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.874157 18279 raft_consensus.cc:2468] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:01.874998 17883 leader_election.cc:304] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:01.875316 18713 leader_election.cc:290] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.875526 18715 raft_consensus.cc:2804] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.875739 18715 raft_consensus.cc:697] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:01.875825 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet d8750126e2be4861b6a61b34af927f54 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.876216 18715 consensus_queue.cc:237] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.879709 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.063s user 0.005s sys 0.001s Metrics: {"bytes_written":454,"delete_count":0,"lbm_write_time_us":171,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260812 20:04:01.880560 18657 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.880833 18657 ts_tablet_manager.cc:1403] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.002s
I20260812 20:04:01.880878 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.882476 18515 tablet_bootstrap.cc:492] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:01.882891 18515 ts_tablet_manager.cc:1403] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.001s
I20260812 20:04:01.883901 18657 raft_consensus.cc:359] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.884202 18657 raft_consensus.cc:385] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.884430 18657 raft_consensus.cc:740] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.884963 18767 tablet_replica.cc:333] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:01.884882 18657 consensus_queue.cc:260] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.884589 17770 catalog_manager.cc:5719] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:01.885274 18767 raft_consensus.cc:2243] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.885682 18767 raft_consensus.cc:2272] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.886456 18657 ts_tablet_manager.cc:1434] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260812 20:04:01.886842 18657 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:01.888204 18767 ts_tablet_manager.cc:1916] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.884307 18515 raft_consensus.cc:359] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.891812 18657 tablet_bootstrap.cc:654] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.892156 18515 raft_consensus.cc:385] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.893493 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.072s user 0.002s sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":726,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":182,"lbm_reads_lt_1ms":8,"lbm_write_time_us":9984,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":3,"mutex_wait_us":8565,"reinsert_count":0,"update_count":4}
I20260812 20:04:01.894562 18508 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:01.894797 18508 ts_tablet_manager.cc:1403] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.006s
I20260812 20:04:01.894584 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.401s user 0.039s sys 0.012s Metrics: {"bytes_written":474,"cfile_cache_hit":15,"cfile_cache_hit_bytes":24862,"cfile_cache_miss":37,"cfile_cache_miss_bytes":34682,"cfile_init":10,"delete_count":0,"delta_iterators_relevant":10,"dirs.queue_time_us":66209,"dirs.run_cpu_time_us":1638,"dirs.run_wall_time_us":10620,"drs_written":1,"lbm_read_time_us":8803,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":76,"lbm_write_time_us":32648,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":30,"num_input_rowsets":2,"peak_mem_usage":139412,"reinsert_count":0,"rows_written":71,"thread_start_us":55214,"threads_started":2,"update_count":2}
I20260812 20:04:01.895761 18508 raft_consensus.cc:359] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.896015 18508 raft_consensus.cc:385] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.896059 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:01.896252 18508 raft_consensus.cc:740] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.896557 18515 raft_consensus.cc:740] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.896586 18508 consensus_queue.cc:260] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.897320 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.071s user 0.003s sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":726,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":183,"lbm_reads_lt_1ms":8,"lbm_write_time_us":371,"lbm_writes_lt_1ms":4,"mutex_wait_us":16999,"reinsert_count":0,"update_count":4}
I20260812 20:04:01.898209 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.898562 18508 ts_tablet_manager.cc:1434] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 20:04:01.899057 18508 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.899021 18515 consensus_queue.cc:260] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.899379 18406 raft_consensus.cc:493] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.899590 18406 raft_consensus.cc:515] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.900413 18406 leader_election.cc:290] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.903349 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:01.903796 18508 tablet_bootstrap.cc:654] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.903967 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9590af0a4c9543bfb8f07837c115ef60" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:01.900033 18515 ts_tablet_manager.cc:1434] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.017s user 0.002s sys 0.002s
I20260812 20:04:01.906418 18515 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:01.906942 18620 raft_consensus.cc:493] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:01.907198 18620 raft_consensus.cc:515] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.908380 18620 leader_election.cc:290] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:01.909811 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5743e1151ef0487ab67f6478b6126279" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:01.910216 18127 raft_consensus.cc:2468] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:01.911208 18208 leader_election.cc:304] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:01.912535 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5743e1151ef0487ab67f6478b6126279" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:01.913352 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9590af0a4c9543bfb8f07837c115ef60" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:01.913712 18279 raft_consensus.cc:2468] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
W20260812 20:04:01.914449 18043 leader_election.cc:343] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.914825 18765 ts_tablet_manager.cc:1929] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:01.916759 18765 log.cc:1199] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/f0229dd61d8b419ca2c4ecbf1940d6d2
I20260812 20:04:01.917644 18765 ts_tablet_manager.cc:1950] T f0229dd61d8b419ca2c4ecbf1940d6d2 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
W20260812 20:04:01.917726 18207 leader_election.cc:343] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.920141 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet f0229dd61d8b419ca2c4ecbf1940d6d2 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.920815 18043 leader_election.cc:304] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:01.916508 18515 tablet_bootstrap.cc:654] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.922596 18620 raft_consensus.cc:2804] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.922298 18765 tablet_replica.cc:333] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.922887 18620 raft_consensus.cc:493] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.923080 18620 raft_consensus.cc:3060] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.926745 18765 raft_consensus.cc:2243] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.927129 18765 raft_consensus.cc:2272] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.926200 18767 ts_tablet_manager.cc:1929] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.927631 18767 log.cc:1199] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/006049a466af4197a4f71989bf9e1b50
I20260812 20:04:01.928395 18767 ts_tablet_manager.cc:1950] T 006049a466af4197a4f71989bf9e1b50 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.931658 18765 ts_tablet_manager.cc:1916] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.934693 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":38062}
I20260812 20:04:01.934646 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.065s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":64795}
I20260812 20:04:01.938979 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 006049a466af4197a4f71989bf9e1b50 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.943116 18620 raft_consensus.cc:515] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.943796 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.062s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":62314}
I20260812 20:04:01.945067 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5743e1151ef0487ab67f6478b6126279" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:01.945677 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5743e1151ef0487ab67f6478b6126279" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:01.946178 18127 raft_consensus.cc:3060] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.946681 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.076s user 0.002s sys 0.006s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":421,"lbm_writes_lt_1ms":4,"mutex_wait_us":66109,"reinsert_count":0,"update_count":1}
I20260812 20:04:01.947408 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.077s user 0.003s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":221,"lbm_writes_lt_1ms":4,"mutex_wait_us":24927,"reinsert_count":0,"update_count":1}
I20260812 20:04:01.948046 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.079s user 0.003s sys 0.005s Metrics: {"bytes_written":461,"delete_count":0,"lbm_write_time_us":228,"lbm_writes_lt_1ms":4,"mutex_wait_us":28119,"reinsert_count":0,"update_count":2}
I20260812 20:04:01.949215 18767 tablet_replica.cc:333] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
W20260812 20:04:01.951107 18207 leader_election.cc:343] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:01.951743 18767 raft_consensus.cc:2243] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.952105 18767 raft_consensus.cc:2272] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:01.952808 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.054s user 0.010s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":780,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":193,"lbm_reads_lt_1ms":8,"lbm_write_time_us":257,"lbm_writes_lt_1ms":4,"mutex_wait_us":36368,"reinsert_count":0,"update_count":6}
I20260812 20:04:01.953919 18724 consensus_queue.cc:1048] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:01.954980 18767 ts_tablet_manager.cc:1916] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:01.961632 18127 raft_consensus.cc:2468] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:01.963699 18208 leader_election.cc:304] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:01.964323 18724 raft_consensus.cc:2804] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:01.970572 18767 ts_tablet_manager.cc:1929] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.970199 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.009s sys 0.001s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":1090,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":77,"lbm_read_time_us":439,"lbm_reads_lt_1ms":14,"lbm_write_time_us":281,"lbm_writes_lt_1ms":4,"mutex_wait_us":14519,"reinsert_count":0,"update_count":6}
I20260812 20:04:01.970901 18767 log.cc:1199] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/8b30713b37a9473395eb907e16e77c7c
I20260812 20:04:01.971875 18767 ts_tablet_manager.cc:1950] T 8b30713b37a9473395eb907e16e77c7c P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:01.973348 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 8b30713b37a9473395eb907e16e77c7c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.974550 18508 tablet_bootstrap.cc:492] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
W20260812 20:04:01.974545 17750 catalog_manager.cc:5017] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): delete failed for tablet 4a1404f2da764dfd8797dfaad98e954f because the tablet was not found. No further retry: Not found: Tablet not found: 4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:01.974937 18508 ts_tablet_manager.cc:1403] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.076s user 0.023s sys 0.046s
I20260812 20:04:01.976032 18508 raft_consensus.cc:359] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.976166 18620 leader_election.cc:290] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:01.976359 18508 raft_consensus.cc:385] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:01.976508 18508 raft_consensus.cc:740] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:01.976853 18811 raft_consensus.cc:2804] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:01.977138 18811 raft_consensus.cc:493] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:01.977034 18508 consensus_queue.cc:260] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.977701 18811 raft_consensus.cc:3060] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.979362 18508 ts_tablet_manager.cc:1434] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 20:04:01.980068 18508 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:01.982522 18765 ts_tablet_manager.cc:1929] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:01.982689 18765 log.cc:1199] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5aac4ecab28843558e26507a083afd4e
I20260812 20:04:01.983749 18765 ts_tablet_manager.cc:1950] T 5aac4ecab28843558e26507a083afd4e P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:01.985044 18765 tablet_replica.cc:333] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:01.985706 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5aac4ecab28843558e26507a083afd4e (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:01.986052 18508 tablet_bootstrap.cc:654] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:01.990603 18811 raft_consensus.cc:515] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.992794 18657 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:01.993498 18811 leader_election.cc:290] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:01.994319 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9590af0a4c9543bfb8f07837c115ef60" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:01.994670 18657 ts_tablet_manager.cc:1403] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.003s
I20260812 20:04:01.994844 17961 raft_consensus.cc:3060] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.995314 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9590af0a4c9543bfb8f07837c115ef60" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:01.995556 18279 raft_consensus.cc:3060] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:01.995995 18657 raft_consensus.cc:359] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:01.994140 18724 raft_consensus.cc:697] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:01.999259 18765 raft_consensus.cc:2243] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:01.999476 17961 raft_consensus.cc:2468] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:01.999583 18765 raft_consensus.cc:2272] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:02.001485 18765 ts_tablet_manager.cc:1916] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:02.004904 18724 consensus_queue.cc:237] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.006541 18043 leader_election.cc:304] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.006747 18657 raft_consensus.cc:385] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.006914 18657 raft_consensus.cc:740] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.007025 18811 raft_consensus.cc:2804] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.007354 18657 consensus_queue.cc:260] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.007503 18811 raft_consensus.cc:697] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:02.008119 18657 ts_tablet_manager.cc:1434] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20260812 20:04:02.008085 18811 consensus_queue.cc:237] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.008515 18657 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.014074 18508 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:02.014273 18508 ts_tablet_manager.cc:1403] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.008s
I20260812 20:04:02.015391 18508 raft_consensus.cc:359] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.015770 18508 raft_consensus.cc:385] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.016028 18508 raft_consensus.cc:740] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.016489 18508 consensus_queue.cc:260] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.019778 18765 ts_tablet_manager.cc:1929] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:02.020032 18508 ts_tablet_manager.cc:1434] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 20:04:02.020499 18508 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:02.020921 18765 log.cc:1199] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b2328bed2bc24cf29b316636d7f493ba
I20260812 20:04:02.021807 18765 ts_tablet_manager.cc:1950] T b2328bed2bc24cf29b316636d7f493ba P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:02.025715 18508 tablet_bootstrap.cc:654] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.027680 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b2328bed2bc24cf29b316636d7f493ba (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:02.031721 18765 tablet_replica.cc:333] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:02.032948 18765 raft_consensus.cc:2243] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
W20260812 20:04:02.034391 18043 consensus_peers.cc:597] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b30713b37a9473395eb907e16e77c7c. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:02.036185 18765 raft_consensus.cc:2272] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:02.038575 18765 ts_tablet_manager.cc:1916] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:02.041260 18515 tablet_bootstrap.cc:492] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.043131 18515 ts_tablet_manager.cc:1403] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.137s user 0.006s sys 0.006s
I20260812 20:04:02.045054 18515 raft_consensus.cc:359] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.047500 18279 raft_consensus.cc:2468] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:02.050652 18515 raft_consensus.cc:385] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.051127 18657 tablet_bootstrap.cc:654] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.052516 18515 raft_consensus.cc:740] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.053083 18515 consensus_queue.cc:260] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.054782 18515 ts_tablet_manager.cc:1434] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20260812 20:04:02.055279 18515 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.061792 18713 raft_consensus.cc:493] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.062038 18713 raft_consensus.cc:515] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.061985 18724 consensus_queue.cc:1048] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.063295 18713 leader_election.cc:290] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.065806 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af421db565214a209958d4ab416c71e3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.066246 17767 catalog_manager.cc:5719] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
W20260812 20:04:02.067057 17884 leader_election.cc:343] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.075148 18715 raft_consensus.cc:493] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.075405 18715 raft_consensus.cc:515] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.076236 18715 leader_election.cc:290] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.076401 18765 ts_tablet_manager.cc:1929] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:02.077839 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.078147 18277 raft_consensus.cc:2468] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:02.079025 17883 leader_election.cc:304] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.079677 18715 raft_consensus.cc:2804] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.079908 18715 raft_consensus.cc:493] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.080224 18715 raft_consensus.cc:3060] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.081501 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.085445 18715 raft_consensus.cc:515] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.077479 18515 tablet_bootstrap.cc:654] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.078703 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.086669 18657 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.086906 18657 ts_tablet_manager.cc:1403] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.006s
W20260812 20:04:02.087553 17884 leader_election.cc:343] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.088035 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:02.088409 18277 raft_consensus.cc:3060] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.088310 18657 raft_consensus.cc:359] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.088678 18657 raft_consensus.cc:385] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.088785 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:02.088932 18765 log.cc:1199] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/8b30713b37a9473395eb907e16e77c7c
I20260812 20:04:02.088948 18657 raft_consensus.cc:740] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.089572 18765 ts_tablet_manager.cc:1950] T 8b30713b37a9473395eb907e16e77c7c P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:02.089696 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.090204 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.090261 18715 leader_election.cc:290] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.090728 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.091279 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.092682 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.093214 18765 tablet_replica.cc:333] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:02.093286 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.093544 18765 raft_consensus.cc:2243] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:02.093647 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.093952 18765 raft_consensus.cc:2272] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:02.094247 18277 raft_consensus.cc:2468] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:02.095000 18765 ts_tablet_manager.cc:1916] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:02.095104 17883 leader_election.cc:304] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.095484 18715 raft_consensus.cc:2804] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.096261 18715 raft_consensus.cc:697] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:02.096666 18715 consensus_queue.cc:237] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:04:02.096871 17884 leader_election.cc:343] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.102739 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.103322 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af421db565214a209958d4ab416c71e3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.104523 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.104645 18838 consensus_queue.cc:1048] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.105182 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.106228 18811 raft_consensus.cc:493] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.106503 18811 raft_consensus.cc:515] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.107501 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.107787 18811 leader_election.cc:290] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.108233 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.108752 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.108903 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f922ded62e854f609412e537f21eb9b1" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.109329 17964 raft_consensus.cc:2468] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:02.109326 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.110049 18043 leader_election.cc:304] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.110481 18811 raft_consensus.cc:2804] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.110798 18811 raft_consensus.cc:493] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.111006 18811 raft_consensus.cc:3060] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.111334 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 8b30713b37a9473395eb907e16e77c7c (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:02.112432 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.116139 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f922ded62e854f609412e537f21eb9b1" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.116468 18811 raft_consensus.cc:515] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.116572 18277 raft_consensus.cc:2468] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:02.117149 17770 catalog_manager.cc:5719] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.118558 17964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f922ded62e854f609412e537f21eb9b1" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:02.118867 17964 raft_consensus.cc:3060] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.121382 18406 raft_consensus.cc:493] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.121713 18406 raft_consensus.cc:515] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.123037 18406 leader_election.cc:290] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.123327 17964 raft_consensus.cc:2468] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:02.125106 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.125036 17959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5128231e073f4f5c8bd9697a9c23a82f" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.125350 18043 leader_election.cc:304] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.125548 18281 raft_consensus.cc:2393] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d222b63b24184ca6bc530a1547fe3297 in current term 1: Already voted for candidate f20226c1597d481a84f0c2fe53f0bb65 in this term.
I20260812 20:04:02.126085 18406 raft_consensus.cc:2804] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.126430 18811 leader_election.cc:290] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.126495 18406 raft_consensus.cc:697] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:02.127056 18406 consensus_queue.cc:237] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.127501 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f922ded62e854f609412e537f21eb9b1" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:02.127856 18281 raft_consensus.cc:3060] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.132066 18281 raft_consensus.cc:2468] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:02.133526 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.134572 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.134575 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.135324 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.147768 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.004s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":251,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.149936 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.004s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":272,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.151335 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.152455 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.175640 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":39688}
I20260812 20:04:02.176630 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.042s user 0.005s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":249,"lbm_writes_lt_1ms":4,"mutex_wait_us":25759,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.177588 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":24375}
I20260812 20:04:02.184701 17770 catalog_manager.cc:5719] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.186792 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":51746}
I20260812 20:04:02.187310 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.078s user 0.007s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":815,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":84,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":744,"lbm_read_time_us":242,"lbm_reads_lt_1ms":8,"lbm_write_time_us":275,"lbm_writes_lt_1ms":4,"mutex_wait_us":30680,"reinsert_count":0,"update_count":7}
I20260812 20:04:02.188261 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.188651 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.188997 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.194871 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.195369 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.196156 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.196707 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.197317 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.201404 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.215864 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.063s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":62555}
I20260812 20:04:02.217783 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.084s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":286,"lbm_writes_lt_1ms":4,"mutex_wait_us":41178,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.218475 18657 consensus_queue.cc:260] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.219267 18279 raft_consensus.cc:2468] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:02.220933 17883 leader_election.cc:304] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.221918 18715 raft_consensus.cc:2804] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.222100 18715 raft_consensus.cc:493] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.222245 18715 raft_consensus.cc:3060] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.225071 18657 ts_tablet_manager.cc:1434] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.138s user 0.003s sys 0.000s
I20260812 20:04:02.227936 18715 raft_consensus.cc:515] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.230562 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af421db565214a209958d4ab416c71e3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:02.230994 18279 raft_consensus.cc:3060] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.231674 18043 leader_election.cc:304] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:02.233604 18657 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.236346 18279 raft_consensus.cc:2468] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:02.237301 17883 leader_election.cc:304] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
W20260812 20:04:02.237521 18043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.14.3:34495, user_credentials={real_user=slave}} blocked reactor thread for 102975us
I20260812 20:04:02.237757 18713 raft_consensus.cc:2804] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.237996 18126 raft_consensus.cc:3060] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.238111 18713 raft_consensus.cc:697] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:02.238535 18481 raft_consensus.cc:3060] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.238510 18713 consensus_queue.cc:237] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.240180 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.133s user 0.009s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":815,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":7716,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":3521,"lbm_read_time_us":277,"lbm_reads_lt_1ms":8,"lbm_write_time_us":552,"lbm_writes_lt_1ms":4,"mutex_wait_us":28623,"reinsert_count":0,"update_count":7}
I20260812 20:04:02.240780 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af421db565214a209958d4ab416c71e3" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:02.240826 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.006s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":280,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.241848 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.242383 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.243216 18657 tablet_bootstrap.cc:654] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.243217 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.247713 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.250284 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.254045 18508 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
W20260812 20:04:02.254434 17884 leader_election.cc:343] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.254920 18508 ts_tablet_manager.cc:1403] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.234s user 0.095s sys 0.076s
I20260812 20:04:02.256810 18724 consensus_queue.cc:1048] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.257766 18508 raft_consensus.cc:359] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.258108 18508 raft_consensus.cc:385] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.258188 18508 raft_consensus.cc:740] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.257993 18715 leader_election.cc:290] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.258841 18508 consensus_queue.cc:260] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.259598 18508 ts_tablet_manager.cc:1434] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 20:04:02.260090 18508 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:02.262233 18515 tablet_bootstrap.cc:492] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.263095 18508 tablet_bootstrap.cc:654] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.264809 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.062s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":62034}
I20260812 20:04:02.265574 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.048s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":47440}
I20260812 20:04:02.266647 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.064s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":246,"lbm_writes_lt_1ms":4,"mutex_wait_us":54270,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.269766 18481 raft_consensus.cc:2749] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.272207 17770 catalog_manager.cc:5719] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.272517 18515 ts_tablet_manager.cc:1403] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.207s user 0.045s sys 0.060s
I20260812 20:04:02.273754 18515 raft_consensus.cc:359] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.274008 18515 raft_consensus.cc:385] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.274183 18515 raft_consensus.cc:740] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.275029 18765 ts_tablet_manager.cc:1929] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:02.275331 18765 log.cc:1199] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/006049a466af4197a4f71989bf9e1b50
I20260812 20:04:02.275851 18765 ts_tablet_manager.cc:1950] T 006049a466af4197a4f71989bf9e1b50 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:02.274760 18515 consensus_queue.cc:260] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.276890 18765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 98) took 1119 ms. Trace:
I20260812 20:04:02.277149 18765 rpcz_store.cc:276] 0812 20:04:01.157712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:01.157776 (+ 64us) service_pool.cc:224] Handling call
0812 20:04:02.093103 (+935327us) ts_tablet_manager.cc:1151] Deleting tablet 006049a466af4197a4f71989bf9e1b50
0812 20:04:02.093120 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:02.236892 (+143772us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:02.272335 (+ 35443us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:02.276738 (+ 4403us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":935089}
I20260812 20:04:02.277871 18765 tablet_replica.cc:333] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:02.277940 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 006049a466af4197a4f71989bf9e1b50 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:02.278206 18765 raft_consensus.cc:2243] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:02.278690 18765 raft_consensus.cc:2272] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:02.279398 18765 ts_tablet_manager.cc:1916] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:02.288542 18814 consensus_queue.cc:1048] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.289887 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.290345 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.290657 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.291407 18715 consensus_queue.cc:1048] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.283607 18515 ts_tablet_manager.cc:1434] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260812 20:04:02.288327 18127 raft_consensus.cc:3060] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.294227 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.294634 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.299302 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.009s sys 0.001s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":1174,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":195,"lbm_read_time_us":349,"lbm_reads_lt_1ms":10,"lbm_write_time_us":318,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":9}
I20260812 20:04:02.299721 18814 raft_consensus.cc:493] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.300180 18814 raft_consensus.cc:515] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.302230 18814 leader_election.cc:290] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.302362 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.113s user 0.005s sys 0.007s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":839,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2865,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":2278,"lbm_read_time_us":326,"lbm_reads_lt_1ms":12,"lbm_write_time_us":232,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":8}
I20260812 20:04:02.303336 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:04:02.304620 17883 leader_election.cc:343] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.304934 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.305330 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.305727 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260812 20:04:02.302141 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.307034 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.307370 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.307600 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.307837 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.307931 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.003s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":234,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.309782 18814 consensus_queue.cc:1048] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 20:04:02.315117 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.316182 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.317837 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.318316 18515 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
W20260812 20:04:02.319288 17884 leader_election.cc:343] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.319585 17884 leader_election.cc:304] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.320542 18713 raft_consensus.cc:2749] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.321547 18814 consensus_queue.cc:1048] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.316395 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.324090 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.324599 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.315002 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":252,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.327131 18515 tablet_bootstrap.cc:654] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.328786 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.329262 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.368408 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.062s user 0.002s sys 0.009s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":1211,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":736,"lbm_read_time_us":286,"lbm_reads_lt_1ms":10,"lbm_write_time_us":338,"lbm_writes_lt_1ms":4,"mutex_wait_us":8661,"reinsert_count":0,"update_count":10}
I20260812 20:04:02.367712 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.043s user 0.007s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":903,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2986,"lbm_read_time_us":186,"lbm_reads_lt_1ms":8,"lbm_write_time_us":11679,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3,"reinsert_count":0,"update_count":10}
I20260812 20:04:02.370707 18657 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.371068 18657 ts_tablet_manager.cc:1403] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.138s user 0.016s sys 0.008s
I20260812 20:04:02.372661 18657 raft_consensus.cc:359] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.373060 18657 raft_consensus.cc:385] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.373265 18657 raft_consensus.cc:740] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.377827 18657 consensus_queue.cc:260] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.378846 18657 ts_tablet_manager.cc:1434] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260812 20:04:02.386533 18657 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.390785 18657 tablet_bootstrap.cc:654] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.398789 18508 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:02.399123 18508 ts_tablet_manager.cc:1403] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.012s
I20260812 20:04:02.400673 18508 raft_consensus.cc:359] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.401006 18508 raft_consensus.cc:385] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.401219 18508 raft_consensus.cc:740] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.402340 18508 consensus_queue.cc:260] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.404214 18508 ts_tablet_manager.cc:1434] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260812 20:04:02.404625 18508 tablet_bootstrap.cc:492] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:02.412974 17767 catalog_manager.cc:5719] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.413704 18124 raft_consensus.cc:3060] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.428821 18860 raft_consensus.cc:493] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.428898 18657 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.429111 18657 ts_tablet_manager.cc:1403] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.007s
I20260812 20:04:02.429100 18860 raft_consensus.cc:515] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.430248 18860 leader_election.cc:290] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.430840 18657 raft_consensus.cc:359] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.431202 18657 raft_consensus.cc:385] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.431396 18657 raft_consensus.cc:740] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.431860 18657 consensus_queue.cc:260] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.432216 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.433941 18657 ts_tablet_manager.cc:1434] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260812 20:04:02.433950 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.434445 18657 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.434468 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.434873 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.435544 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.435974 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.436576 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.436023 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":11008}
I20260812 20:04:02.436501 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.439468 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.439982 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.441133 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.441534 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.441900 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.442451 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.442680 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.452701 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.453909 18657 tablet_bootstrap.cc:654] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.456506 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":348,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.457376 18765 ts_tablet_manager.cc:1929] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:02.457695 18481 consensus_queue.cc:1048] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 20:04:02.458197 18765 log.cc:1199] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/d8750126e2be4861b6a61b34af927f54
I20260812 20:04:02.460355 18724 raft_consensus.cc:493] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.460634 18724 raft_consensus.cc:515] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.461771 18724 leader_election.cc:290] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:02.462895 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:02.463685 18208 leader_election.cc:343] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.463960 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.031s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":267,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
W20260812 20:04:02.465472 17884 consensus_peers.cc:597] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:02.465647 18765 ts_tablet_manager.cc:1950] T d8750126e2be4861b6a61b34af927f54 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:02.465432 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.009s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":925,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":8728,"lbm_read_time_us":220,"lbm_reads_lt_1ms":8,"lbm_write_time_us":262,"lbm_writes_lt_1ms":4,"mutex_wait_us":6801,"reinsert_count":0,"update_count":11}
I20260812 20:04:02.466874 18765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 97) took 1309 ms. Trace:
I20260812 20:04:02.467067 18765 rpcz_store.cc:276] 0812 20:04:01.157537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:01.157949 (+ 412us) service_pool.cc:224] Handling call
0812 20:04:02.277515 (+1119566us) ts_tablet_manager.cc:1151] Deleting tablet d8750126e2be4861b6a61b34af927f54
0812 20:04:02.277529 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:02.417563 (+140034us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:02.452492 (+ 34929us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:02.466815 (+ 14323us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119390}
I20260812 20:04:02.467729 18765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 99) took 1307 ms. Trace:
I20260812 20:04:02.467808 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet d8750126e2be4861b6a61b34af927f54 (table table-2 [id=cab60ba0b5e7477b9b08d4592a5c6bc3]) successfully deleted
I20260812 20:04:02.468010 18765 rpcz_store.cc:276] 0812 20:04:01.160216 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:01.160649 (+ 433us) service_pool.cc:224] Handling call
0812 20:04:02.467465 (+1306816us) ts_tablet_manager.cc:1151] Deleting tablet 4a1404f2da764dfd8797dfaad98e954f
0812 20:04:02.467477 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:02.467613 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306568}
W20260812 20:04:02.468389 17884 leader_election.cc:343] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:02.469893 17751 catalog_manager.cc:5017] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): delete failed for tablet 4a1404f2da764dfd8797dfaad98e954f because the tablet was not found. No further retry: Not found: Tablet not found: 4a1404f2da764dfd8797dfaad98e954f
I20260812 20:04:02.470577 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:04:02.470708 18207 consensus_peers.cc:597] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:02.470899 17961 raft_consensus.cc:2468] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:02.471299 18860 consensus_queue.cc:1048] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260812 20:04:02.472838 18508 tablet_bootstrap.cc:654] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.473768 18207 leader_election.cc:304] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.474498 18515 tablet_bootstrap.cc:492] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.474728 18515 ts_tablet_manager.cc:1403] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.179s user 0.008s sys 0.034s
I20260812 20:04:02.476052 18515 raft_consensus.cc:359] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.476238 18515 raft_consensus.cc:385] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.476338 18515 raft_consensus.cc:740] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.476570 18515 consensus_queue.cc:260] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.477952 18515 ts_tablet_manager.cc:1434] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:04:02.478611 18515 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.485707 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.043s user 0.006s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":925,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2726,"lbm_read_time_us":208,"lbm_reads_lt_1ms":8,"lbm_write_time_us":212,"lbm_writes_lt_1ms":4,"mutex_wait_us":9941,"reinsert_count":0,"update_count":11}
I20260812 20:04:02.487373 18515 tablet_bootstrap.cc:654] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.494323 18724 raft_consensus.cc:2804] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260812 20:04:02.499830 17884 consensus_peers.cc:597] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:02.502084 18515 tablet_bootstrap.cc:492] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.502467 18515 ts_tablet_manager.cc:1403] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.002s
I20260812 20:04:02.503229 18508 tablet_bootstrap.cc:492] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:02.503593 18508 ts_tablet_manager.cc:1403] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.003s
I20260812 20:04:02.503846 18515 raft_consensus.cc:359] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.504068 18515 raft_consensus.cc:385] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.504144 18515 raft_consensus.cc:740] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.504383 18515 consensus_queue.cc:260] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.505349 18860 consensus_queue.cc:1048] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.506547 18860 consensus_queue.cc:1048] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.507339 18515 ts_tablet_manager.cc:1434] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260812 20:04:02.509424 18515 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.512068 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.513577 18724 raft_consensus.cc:493] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.513578 18657 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.513825 18724 raft_consensus.cc:3060] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.514380 18657 ts_tablet_manager.cc:1403] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.004s
W20260812 20:04:02.515190 17883 leader_election.cc:343] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.515513 17883 leader_election.cc:304] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.516052 18860 raft_consensus.cc:2749] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.516477 18508 raft_consensus.cc:359] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.516848 18508 raft_consensus.cc:385] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.517038 18508 raft_consensus.cc:740] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.517495 18508 consensus_queue.cc:260] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.524304 18508 ts_tablet_manager.cc:1434] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.020s user 0.000s sys 0.004s
I20260812 20:04:02.528275 18508 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:02.532657 18508 tablet_bootstrap.cc:654] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.536531 18657 raft_consensus.cc:359] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.537854 18657 raft_consensus.cc:385] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.538069 18657 raft_consensus.cc:740] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.540038 18657 consensus_queue.cc:260] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.541088 18657 ts_tablet_manager.cc:1434] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.026s user 0.000s sys 0.004s
I20260812 20:04:02.541736 18657 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.542898 18724 raft_consensus.cc:515] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.547233 18515 tablet_bootstrap.cc:654] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.552505 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:02.552556 18724 leader_election.cc:290] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:02.552878 17961 raft_consensus.cc:3060] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.553238 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:02.556891 18657 tablet_bootstrap.cc:654] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
W20260812 20:04:02.557469 18208 leader_election.cc:343] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.557993 18860 consensus_queue.cc:1048] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.562878 17961 raft_consensus.cc:2468] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:02.573733 18207 leader_election.cc:304] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.574379 18832 raft_consensus.cc:2804] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.574616 18832 raft_consensus.cc:697] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:02.575214 18832 consensus_queue.cc:237] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.581399 18481 consensus_queue.cc:1048] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260812 20:04:02.581939 18515 tablet_bootstrap.cc:492] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.582617 18515 ts_tablet_manager.cc:1403] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.006s
I20260812 20:04:02.583938 18515 raft_consensus.cc:359] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.584208 18515 raft_consensus.cc:385] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.584652 18515 raft_consensus.cc:740] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.585332 18515 consensus_queue.cc:260] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.590899 18515 ts_tablet_manager.cc:1434] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260812 20:04:02.595073 18515 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.605634 18657 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.605898 18657 ts_tablet_manager.cc:1403] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.017s
I20260812 20:04:02.607327 18657 raft_consensus.cc:359] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.607590 18657 raft_consensus.cc:385] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.607755 18657 raft_consensus.cc:740] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.608326 18657 consensus_queue.cc:260] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.609254 18657 ts_tablet_manager.cc:1434] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260812 20:04:02.609706 18657 tablet_bootstrap.cc:492] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:02.610015 18508 tablet_bootstrap.cc:492] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:02.610327 18508 ts_tablet_manager.cc:1403] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.011s
I20260812 20:04:02.611801 18508 raft_consensus.cc:359] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.612259 18508 raft_consensus.cc:385] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.612530 18508 raft_consensus.cc:740] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.612859 18508 consensus_queue.cc:260] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.613404 18657 tablet_bootstrap.cc:654] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.616465 18515 tablet_bootstrap.cc:654] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.618909 18811 consensus_queue.cc:1048] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.624238 18838 consensus_queue.cc:1048] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.619450 17770 catalog_manager.cc:5719] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:02.627437 18508 ts_tablet_manager.cc:1434] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20260812 20:04:02.630332 18508 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:02.630412 18657 tablet_bootstrap.cc:492] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:02.630755 18657 ts_tablet_manager.cc:1403] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20260812 20:04:02.632270 18657 raft_consensus.cc:359] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.632689 18657 raft_consensus.cc:385] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.632856 18657 raft_consensus.cc:740] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.633584 18508 tablet_bootstrap.cc:654] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.640699 18657 consensus_queue.cc:260] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.642598 18657 ts_tablet_manager.cc:1434] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.012s user 0.000s sys 0.004s
I20260812 20:04:02.645354 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.645879 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.646310 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.646225 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.646670 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.646889 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.647342 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.647706 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.648152 18724 raft_consensus.cc:493] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.648401 18724 raft_consensus.cc:515] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.649425 18724 leader_election.cc:290] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:02.649912 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.650332 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.650579 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.651796 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.652366 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.652824 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.654928 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.655126 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20260812 20:04:02.656400 18208 leader_election.cc:343] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.656291 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:04:02.657366 18207 leader_election.cc:343] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.657639 18207 leader_election.cc:304] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:02.658373 18724 raft_consensus.cc:2749] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.662773 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.663290 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.663971 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.678817 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.004s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":205,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.679059 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.005s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":345,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.679824 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":953,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":788,"lbm_read_time_us":195,"lbm_reads_lt_1ms":8,"lbm_write_time_us":244,"lbm_writes_lt_1ms":4,"mutex_wait_us":3725,"reinsert_count":0,"update_count":12}
I20260812 20:04:02.681419 18515 tablet_bootstrap.cc:492] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.681859 18515 ts_tablet_manager.cc:1403] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.003s
I20260812 20:04:02.683320 18860 raft_consensus.cc:493] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.683660 18860 raft_consensus.cc:515] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.685129 18860 leader_election.cc:290] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.685899 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4581755097204339b4b41750e54ac482" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.685928 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4581755097204339b4b41750e54ac482" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.686439 18277 raft_consensus.cc:2468] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
W20260812 20:04:02.687073 17884 leader_election.cc:343] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.688004 17883 leader_election.cc:304] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.690992 18860 raft_consensus.cc:2804] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.691421 18860 raft_consensus.cc:493] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.691572 18860 raft_consensus.cc:3060] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.692590 18406 consensus_queue.cc:1048] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.704664 18860 raft_consensus.cc:515] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.706879 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4581755097204339b4b41750e54ac482" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
W20260812 20:04:02.707990 17884 leader_election.cc:343] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:02.708761 18406 raft_consensus.cc:493] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.709076 18406 raft_consensus.cc:515] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.710232 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4581755097204339b4b41750e54ac482" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:02.710408 18406 leader_election.cc:290] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.710640 18281 raft_consensus.cc:3060] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.711689 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.712921 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de0311de0f984639b73eddfef7076109" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.714686 18043 leader_election.cc:304] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:02.715361 18406 raft_consensus.cc:2749] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.686221 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.007s sys 0.006s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":953,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2106,"lbm_read_time_us":272,"lbm_reads_lt_1ms":8,"lbm_write_time_us":245,"lbm_writes_lt_1ms":4,"mutex_wait_us":7447,"reinsert_count":0,"update_count":12}
I20260812 20:04:02.717698 18515 raft_consensus.cc:359] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.718200 18515 raft_consensus.cc:385] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.718314 18515 raft_consensus.cc:740] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.720300 18515 consensus_queue.cc:260] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.721096 18515 ts_tablet_manager.cc:1434] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.039s user 0.003s sys 0.000s
I20260812 20:04:02.721652 18515 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.723708 18508 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:02.724032 18508 ts_tablet_manager.cc:1403] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.032s
I20260812 20:04:02.725001 18508 raft_consensus.cc:359] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.725078 18281 raft_consensus.cc:2468] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:02.725224 18508 raft_consensus.cc:385] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.725381 18508 raft_consensus.cc:740] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.726114 17883 leader_election.cc:304] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:02.726743 18814 raft_consensus.cc:2804] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.726963 18814 raft_consensus.cc:697] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:02.727339 18860 leader_election.cc:290] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.727497 18814 consensus_queue.cc:237] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.728356 18508 consensus_queue.cc:260] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.729965 18515 tablet_bootstrap.cc:654] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.731508 18508 ts_tablet_manager.cc:1434] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260812 20:04:02.736411 17767 catalog_manager.cc:5719] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.739531 18515 tablet_bootstrap.cc:492] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.739730 18515 ts_tablet_manager.cc:1403] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20260812 20:04:02.740978 18515 raft_consensus.cc:359] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.741356 18515 raft_consensus.cc:385] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.742205 18515 raft_consensus.cc:740] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.742595 18515 consensus_queue.cc:260] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.745132 18515 ts_tablet_manager.cc:1434] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260812 20:04:02.745582 18515 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.748987 18406 raft_consensus.cc:493] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.749202 18406 raft_consensus.cc:515] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.750392 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.750598 18406 leader_election.cc:290] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.750775 18515 tablet_bootstrap.cc:654] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.750820 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.750900 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.750985 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.751199 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.751358 18281 raft_consensus.cc:2468] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:02.751358 17962 raft_consensus.cc:2468] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:02.751616 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.751935 18043 leader_election.cc:304] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.752140 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.752586 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.752734 18406 raft_consensus.cc:2804] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.753046 18406 raft_consensus.cc:493] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.753073 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.753261 18406 raft_consensus.cc:3060] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.754223 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.754778 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.755100 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.755350 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.755591 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.756076 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.756669 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.757103 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.757699 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.758390 18481 raft_consensus.cc:493] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.758774 18481 raft_consensus.cc:515] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.759449 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.004s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":207,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.759927 18481 leader_election.cc:290] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.761807 18406 raft_consensus.cc:515] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.762660 18406 leader_election.cc:290] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.764508 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.764353 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":276,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.764981 17962 raft_consensus.cc:2393] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d222b63b24184ca6bc530a1547fe3297 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:02.765679 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:02.765952 17961 raft_consensus.cc:3060] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.767030 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7576613d620548df8dbd661bb74dca45" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:02.767122 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d9104457094da480d79c4e41fe9f77" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.767390 18277 raft_consensus.cc:3060] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.768110 18043 leader_election.cc:304] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:02.768877 18406 raft_consensus.cc:3060] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.769951 17961 raft_consensus.cc:2468] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:02.770697 18043 leader_election.cc:304] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:02.771181 18481 raft_consensus.cc:2804] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.771396 18481 raft_consensus.cc:697] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:02.771917 18481 consensus_queue.cc:237] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.771778 18277 raft_consensus.cc:2468] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:02.773193 18406 raft_consensus.cc:2749] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:02.775127 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.007s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":977,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":738,"lbm_read_time_us":187,"lbm_reads_lt_1ms":8,"lbm_write_time_us":226,"lbm_writes_lt_1ms":4,"mutex_wait_us":7043,"reinsert_count":0,"update_count":13}
I20260812 20:04:02.777524 17770 catalog_manager.cc:5719] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:02.779435 18515 tablet_bootstrap.cc:492] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.779781 18515 ts_tablet_manager.cc:1403] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.014s
I20260812 20:04:02.782080 18515 raft_consensus.cc:359] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.781914 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.007s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":977,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":76,"lbm_read_time_us":179,"lbm_reads_lt_1ms":8,"lbm_write_time_us":295,"lbm_writes_lt_1ms":4,"mutex_wait_us":13100,"reinsert_count":0,"update_count":13}
I20260812 20:04:02.782546 18515 raft_consensus.cc:385] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.782747 18515 raft_consensus.cc:740] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.783350 18515 consensus_queue.cc:260] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.784113 18515 ts_tablet_manager.cc:1434] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 20:04:02.784669 18515 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.787879 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.788347 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.789434 18515 tablet_bootstrap.cc:654] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.790073 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.791198 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.791702 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.792520 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.793864 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.794533 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.797398 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.798352 18515 tablet_bootstrap.cc:492] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:02.798350 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.798630 18515 ts_tablet_manager.cc:1403] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260812 20:04:02.799358 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":262,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.800326 18515 raft_consensus.cc:359] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.800640 18515 raft_consensus.cc:385] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:02.801040 18515 raft_consensus.cc:740] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:02.801687 18515 consensus_queue.cc:260] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.802815 18515 ts_tablet_manager.cc:1434] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260812 20:04:02.803464 18515 tablet_bootstrap.cc:492] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:02.804476 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1003,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":66,"lbm_read_time_us":270,"lbm_reads_lt_1ms":8,"lbm_write_time_us":171,"lbm_writes_lt_1ms":4,"mutex_wait_us":3542,"reinsert_count":0,"update_count":14}
I20260812 20:04:02.807224 18515 tablet_bootstrap.cc:654] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:02.810427 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":473,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
W20260812 20:04:02.811704 18515 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260812 20:04:02.812775 18515 ts_tablet_manager.cc:1425] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Tablet failed to bootstrap: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4581755097204339b4b41750e54ac482/.kudutmp.newsegmentWeJUdg (525082624 bytes available vs 518351011 bytes reserved) (error 28)
I20260812 20:04:02.813189 18515 ts_tablet_manager.cc:1403] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 20:04:02.813544 18515 tablet_replica.cc:333] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:02.813768 18515 raft_consensus.cc:2243] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:02.814072 18515 raft_consensus.cc:2272] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:02.817123 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1003,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":113,"lbm_read_time_us":287,"lbm_reads_lt_1ms":8,"lbm_write_time_us":272,"lbm_writes_lt_1ms":4,"mutex_wait_us":7303,"reinsert_count":0,"update_count":14}
I20260812 20:04:02.823180 17961 raft_consensus.cc:3060] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.828536 18814 consensus_queue.cc:1048] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.833412 18838 raft_consensus.cc:493] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.833920 18838 raft_consensus.cc:515] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.835345 18838 leader_election.cc:290] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:02.836110 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:02.836472 18124 raft_consensus.cc:2468] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:02.836504 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.837028 18208 leader_election.cc:304] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:02.837095 17961 raft_consensus.cc:2468] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:02.837357 18838 raft_consensus.cc:2804] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:02.837504 18838 raft_consensus.cc:493] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:02.837649 18838 raft_consensus.cc:3060] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.840826 18124 raft_consensus.cc:3060] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.842746 18838 raft_consensus.cc:515] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.844231 18838 leader_election.cc:290] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:02.844739 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:02.845042 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d8b11ecadeb4e25b4eb9abc97920d95" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:02.845382 17961 raft_consensus.cc:3060] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.845381 18125 raft_consensus.cc:3060] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.848019 18713 consensus_queue.cc:1048] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.850474 18125 raft_consensus.cc:2468] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:02.851320 17961 raft_consensus.cc:2468] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:02.851430 18208 leader_election.cc:304] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:02.852001 18838 raft_consensus.cc:2804] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:02.852241 18838 raft_consensus.cc:697] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:02.853193 18838 consensus_queue.cc:237] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.857764 17767 catalog_manager.cc:5719] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:02.868206 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.868640 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.869051 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.869697 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.870251 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.870841 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:02.871452 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.871938 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.872396 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.872828 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:02.872977 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.873255 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:02.874853 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":208,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.876523 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:02.881856 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.006s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1025,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":62,"lbm_read_time_us":109,"lbm_reads_lt_1ms":8,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"mutex_wait_us":4852,"reinsert_count":0,"update_count":15}
I20260812 20:04:02.882517 18124 raft_consensus.cc:3060] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:02.883636 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.000s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1025,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":134,"lbm_read_time_us":127,"lbm_reads_lt_1ms":8,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"mutex_wait_us":3680,"reinsert_count":0,"update_count":15}
I20260812 20:04:02.956224 18838 consensus_queue.cc:1048] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.964025 18481 raft_consensus.cc:493] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:02.964336 18481 raft_consensus.cc:515] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:02.965231 18481 leader_election.cc:290] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:02.966284 18620 consensus_queue.cc:1048] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:02.970513 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:02.971906 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "568e19899d4140ac8f708e183f2cd8c2" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:02.972786 18043 leader_election.cc:304] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:02.973763 18481 raft_consensus.cc:2749] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:03.027336 18124 raft_consensus.cc:3060] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.040788 18481 raft_consensus.cc:493] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.041127 18481 raft_consensus.cc:515] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.042140 18481 leader_election.cc:290] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.042904 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:03.042908 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:03.043226 18277 raft_consensus.cc:2468] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:03.043251 17961 raft_consensus.cc:2468] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:03.043926 18043 leader_election.cc:304] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:03.044535 18481 raft_consensus.cc:2804] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:03.044731 18481 raft_consensus.cc:493] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:03.044996 18481 raft_consensus.cc:3060] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.048472 18481 raft_consensus.cc:515] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.049364 18481 leader_election.cc:290] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.052224 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:03.052691 17961 raft_consensus.cc:3060] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.052781 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b5eebe47827446aae93c30cdaa4e345" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:03.053140 18277 raft_consensus.cc:3060] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.057153 17961 raft_consensus.cc:2468] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:03.057305 18277 raft_consensus.cc:2468] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:03.058300 18043 leader_election.cc:304] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:03.058929 18481 raft_consensus.cc:2804] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:03.059386 18481 raft_consensus.cc:697] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:03.060110 18481 consensus_queue.cc:237] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.064476 17767 catalog_manager.cc:5719] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:03.075942 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.076385 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.077908 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.078560 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.080575 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.080876 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.081269 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.081805 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.082082 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.082464 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.082795 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.083161 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.083808 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.088060 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":234,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.090219 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.008s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1055,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":144,"lbm_read_time_us":198,"lbm_reads_lt_1ms":8,"lbm_write_time_us":295,"lbm_writes_lt_1ms":4,"mutex_wait_us":2380,"reinsert_count":0,"update_count":16}
I20260812 20:04:03.097577 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.004s sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1055,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":121,"lbm_read_time_us":181,"lbm_reads_lt_1ms":8,"lbm_write_time_us":222,"lbm_writes_lt_1ms":4,"mutex_wait_us":5359,"reinsert_count":0,"update_count":16}
I20260812 20:04:03.178632 17767 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-3" } modify_external_catalogs: true
I20260812 20:04:03.178905 17767 catalog_manager.cc:2781] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38702:
table { table_name: "table-3" } modify_external_catalogs: true
I20260812 20:04:03.188810 18481 consensus_queue.cc:1048] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:03.193826 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.194332 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.194697 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.195602 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.196782 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.197409 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 7576613d620548df8dbd661bb74dca45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.197476 18406 consensus_queue.cc:1048] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:03.199386 18901 tablet_replica.cc:333] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.197441 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet e88eefade65344a4ab4ac5ac3bb9d4ac
I20260812 20:04:03.200829 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 73026c3f231b469887aa3cfadf305ade
I20260812 20:04:03.201263 18256 tablet_service.cc:1558] Processing DeleteTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.201697 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01
I20260812 20:04:03.202478 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b50886c57ca84a5ea6c5e5b1278de7ac
I20260812 20:04:03.202530 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.203064 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5128231e073f4f5c8bd9697a9c23a82f
I20260812 20:04:03.203248 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet 73026c3f231b469887aa3cfadf305ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.203915 17927 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.204203 17922 tablet_service.cc:1558] Processing DeleteTablet for tablet e88eefade65344a4ab4ac5ac3bb9d4ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.204628 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 4d8b11ecadeb4e25b4eb9abc97920d95
I20260812 20:04:03.204661 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 73026c3f231b469887aa3cfadf305ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.204622 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 73026c3f231b469887aa3cfadf305ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.204912 18249 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.204978 17938 tablet_service.cc:1558] Processing DeleteTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.205260 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.205441 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 9590af0a4c9543bfb8f07837c115ef60
I20260812 20:04:03.206131 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet f922ded62e854f609412e537f21eb9b1
I20260812 20:04:03.206393 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 7576613d620548df8dbd661bb74dca45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.206868 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet de0311de0f984639b73eddfef7076109
I20260812 20:04:03.206818 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.207077 18106 tablet_service.cc:1558] Processing DeleteTablet for tablet b50886c57ca84a5ea6c5e5b1278de7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.207556 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet af421db565214a209958d4ab416c71e3
I20260812 20:04:03.207703 18249 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.207818 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.208040 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.208098 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 568e19899d4140ac8f708e183f2cd8c2
I20260812 20:04:03.208241 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.208459 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.209609 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 5743e1151ef0487ab67f6478b6126279
I20260812 20:04:03.210029 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet de0311de0f984639b73eddfef7076109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.210160 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet b6d9104457094da480d79c4e41fe9f77
I20260812 20:04:03.210188 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.210436 18249 tablet_service.cc:1558] Processing DeleteTablet for tablet af421db565214a209958d4ab416c71e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.210522 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f922ded62e854f609412e537f21eb9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.210824 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 2b5eebe47827446aae93c30cdaa4e345
I20260812 20:04:03.210908 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet f922ded62e854f609412e537f21eb9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.210968 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet af421db565214a209958d4ab416c71e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.211158 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet de0311de0f984639b73eddfef7076109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.211408 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.211529 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 4581755097204339b4b41750e54ac482
I20260812 20:04:03.212098 17767 catalog_manager.cc:6006] T 00000000000000000000000000000000 P a3cb6cd752da451f99894b4d53d113d1: Sending DeleteTablet for 3 replicas of tablet 7576613d620548df8dbd661bb74dca45
I20260812 20:04:03.212944 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet af421db565214a209958d4ab416c71e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.213047 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.213549 18257 tablet_service.cc:1558] Processing DeleteTablet for tablet 4581755097204339b4b41750e54ac482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.213251 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.214118 18242 tablet_service.cc:1558] Processing DeleteTablet for tablet b6d9104457094da480d79c4e41fe9f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.214099 17938 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d8b11ecadeb4e25b4eb9abc97920d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.214146 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.214483 18249 tablet_service.cc:1558] Processing DeleteTablet for tablet 5743e1151ef0487ab67f6478b6126279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.214483 17927 tablet_service.cc:1558] Processing DeleteTablet for tablet de0311de0f984639b73eddfef7076109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.214622 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.214797 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.215286 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.215737 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.216107 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.216988 18247 tablet_service.cc:1558] Processing DeleteTablet for tablet 7576613d620548df8dbd661bb74dca45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:43996
I20260812 20:04:03.217482 18106 tablet_service.cc:1558] Processing DeleteTablet for tablet 4581755097204339b4b41750e54ac482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.217914 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.218259 18105 tablet_service.cc:1558] Processing DeleteTablet for tablet b6d9104457094da480d79c4e41fe9f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.218385 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 5743e1151ef0487ab67f6478b6126279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.220184 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 7576613d620548df8dbd661bb74dca45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:40402
I20260812 20:04:03.220572 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.343567
I20260812 20:04:03.224486 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.224781 17922 tablet_service.cc:1558] Processing DeleteTablet for tablet f922ded62e854f609412e537f21eb9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.224874 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.224905 18713 consensus_queue.cc:1048] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260812 20:04:03.225111 17927 tablet_service.cc:1558] Processing DeleteTablet for tablet 5743e1151ef0487ab67f6478b6126279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.225426 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 9590af0a4c9543bfb8f07837c115ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.225827 17920 tablet_service.cc:1558] Processing DeleteTablet for tablet 5128231e073f4f5c8bd9697a9c23a82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.225991 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet b6d9104457094da480d79c4e41fe9f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.226172 17937 tablet_service.cc:1558] Processing DeleteTablet for tablet 568e19899d4140ac8f708e183f2cd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.226449 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b5eebe47827446aae93c30cdaa4e345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.226910 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 4581755097204339b4b41750e54ac482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.227479 17921 tablet_service.cc:1558] Processing DeleteTablet for tablet 7576613d620548df8dbd661bb74dca45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 20:04:03 UTC) from {username='slave'} at 127.0.0.1:44302
I20260812 20:04:03.227833 18901 raft_consensus.cc:2243] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.228822 18901 raft_consensus.cc:2272] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.229836 17770 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38702:
name: "table-4"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260812 20:04:03.230126 18901 ts_tablet_manager.cc:1916] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:04:03.233186 17770 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 20:04:03.234309 18903 tablet_replica.cc:333] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.234704 18904 tablet_replica.cc:333] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.234704 18903 raft_consensus.cc:2243] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.235353 18903 raft_consensus.cc:2272] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.236045 18904 raft_consensus.cc:2243] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.236402 18903 ts_tablet_manager.cc:1916] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.236637 18904 raft_consensus.cc:2272] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.238838 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.044s user 0.005s sys 0.002s Metrics: {"bytes_written":890,"delete_count":0,"lbm_write_time_us":337,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":15}
I20260812 20:04:03.240136 18904 ts_tablet_manager.cc:1916] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.240978 18901 ts_tablet_manager.cc:1929] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.241236 18901 log.cc:1199] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/7576613d620548df8dbd661bb74dca45
I20260812 20:04:03.241554 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.139981
I20260812 20:04:03.242100 18901 ts_tablet_manager.cc:1950] T 7576613d620548df8dbd661bb74dca45 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.243474 18901 tablet_replica.cc:333] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.243867 18901 raft_consensus.cc:2243] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.244406 18901 raft_consensus.cc:2272] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.245591 18901 ts_tablet_manager.cc:1916] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.247355 18903 ts_tablet_manager.cc:1929] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.247680 18903 log.cc:1199] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/e88eefade65344a4ab4ac5ac3bb9d4ac
I20260812 20:04:03.248636 18903 ts_tablet_manager.cc:1950] T e88eefade65344a4ab4ac5ac3bb9d4ac P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.250335 18903 tablet_replica.cc:333] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.250875 18903 raft_consensus.cc:2243] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.251113 18904 ts_tablet_manager.cc:1929] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.251479 18903 raft_consensus.cc:2272] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.252673 18904 log.cc:1199] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/e88eefade65344a4ab4ac5ac3bb9d4ac
I20260812 20:04:03.254364 18903 ts_tablet_manager.cc:1916] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.254647 18904 ts_tablet_manager.cc:1950] T e88eefade65344a4ab4ac5ac3bb9d4ac P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.255160 18901 ts_tablet_manager.cc:1929] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.255261 17884 consensus_queue.cc:579] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: The tablet replica hosted on peer d222b63b24184ca6bc530a1547fe3297 has failed
W20260812 20:04:03.255523 17884 consensus_peers.cc:597] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4581755097204339b4b41750e54ac482/.kudutmp.newsegmentWeJUdg (525082624 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:03.256032 18904 tablet_replica.cc:333] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.255961 18901 log.cc:1199] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/4f48b54fa13d4c1ba18d59e9dd8dbb01
I20260812 20:04:03.256364 18904 raft_consensus.cc:2243] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.256651 18904 raft_consensus.cc:2272] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.257444 18901 ts_tablet_manager.cc:1950] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.258781 18901 tablet_replica.cc:333] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.259120 18901 raft_consensus.cc:2243] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.259550 18901 raft_consensus.cc:2272] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.260475 18901 ts_tablet_manager.cc:1916] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.263911 18904 ts_tablet_manager.cc:1916] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.270040 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":72881}
I20260812 20:04:03.257002 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet e88eefade65344a4ab4ac5ac3bb9d4ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.259145 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet e88eefade65344a4ab4ac5ac3bb9d4ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.271631 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.271967 18904 ts_tablet_manager.cc:1929] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.272287 18904 log.cc:1199] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/73026c3f231b469887aa3cfadf305ade
I20260812 20:04:03.272671 18901 ts_tablet_manager.cc:1929] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.273015 18901 log.cc:1199] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/e88eefade65344a4ab4ac5ac3bb9d4ac
I20260812 20:04:03.273013 18903 ts_tablet_manager.cc:1929] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.273129 18904 ts_tablet_manager.cc:1950] T 73026c3f231b469887aa3cfadf305ade P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.273324 18903 log.cc:1199] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/73026c3f231b469887aa3cfadf305ade
I20260812 20:04:03.273862 18903 ts_tablet_manager.cc:1950] T 73026c3f231b469887aa3cfadf305ade P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.274034 18901 ts_tablet_manager.cc:1950] T e88eefade65344a4ab4ac5ac3bb9d4ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.275092 18903 tablet_replica.cc:333] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.275187 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 73026c3f231b469887aa3cfadf305ade (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.275406 18903 raft_consensus.cc:2243] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.275522 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet e88eefade65344a4ab4ac5ac3bb9d4ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.275589 18901 tablet_replica.cc:333] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.275683 18904 tablet_replica.cc:333] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.275756 18903 raft_consensus.cc:2272] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.275872 18901 raft_consensus.cc:2243] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.275943 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 73026c3f231b469887aa3cfadf305ade (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.276221 18901 raft_consensus.cc:2272] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.276377 18904 raft_consensus.cc:2243] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.276710 18903 ts_tablet_manager.cc:1916] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.276892 18904 raft_consensus.cc:2272] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.277143 18901 ts_tablet_manager.cc:1916] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.277999 18904 ts_tablet_manager.cc:1916] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.270934 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.006s sys 0.001s Metrics: {"bytes_written":936,"delete_count":0,"lbm_write_time_us":207,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260812 20:04:03.281243 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.084s user 0.003s sys 0.004s Metrics: {"bytes_written":471,"delete_count":0,"lbm_write_time_us":223,"lbm_writes_lt_1ms":4,"mutex_wait_us":42047,"reinsert_count":0,"update_count":2}
I20260812 20:04:03.282217 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.085s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":84843}
I20260812 20:04:03.285053 18903 ts_tablet_manager.cc:1929] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.285426 18903 log.cc:1199] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/4f48b54fa13d4c1ba18d59e9dd8dbb01
I20260812 20:04:03.286336 18903 ts_tablet_manager.cc:1950] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.286516 18901 ts_tablet_manager.cc:1929] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.286916 18901 log.cc:1199] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/73026c3f231b469887aa3cfadf305ade
I20260812 20:04:03.287667 18901 ts_tablet_manager.cc:1950] T 73026c3f231b469887aa3cfadf305ade P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.287839 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.288400 18903 tablet_replica.cc:333] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.288818 18903 raft_consensus.cc:2243] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.289179 18903 raft_consensus.cc:2272] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.289441 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 73026c3f231b469887aa3cfadf305ade (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.290764 18903 ts_tablet_manager.cc:1916] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.291930 18901 tablet_replica.cc:333] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.292294 18901 raft_consensus.cc:2243] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.293094 18901 raft_consensus.cc:2272] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.294337 18901 ts_tablet_manager.cc:1916] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.299283 18903 ts_tablet_manager.cc:1929] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.299609 18904 ts_tablet_manager.cc:1929] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.299647 18903 log.cc:1199] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b50886c57ca84a5ea6c5e5b1278de7ac
I20260812 20:04:03.300402 18903 ts_tablet_manager.cc:1950] T b50886c57ca84a5ea6c5e5b1278de7ac P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.301816 18903 tablet_replica.cc:333] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.302179 18903 raft_consensus.cc:2243] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.302138 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.302613 18903 pending_rounds.cc:70] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Trying to abort 1 pending ops.
I20260812 20:04:03.302770 18903 pending_rounds.cc:77] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317770415530622976 op_type: NO_OP noop_request { }
I20260812 20:04:03.304606 18904 log.cc:1199] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/7576613d620548df8dbd661bb74dca45
I20260812 20:04:03.305347 18904 ts_tablet_manager.cc:1950] T 7576613d620548df8dbd661bb74dca45 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.306643 18903 raft_consensus.cc:2889] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260812 20:04:03.306866 18904 tablet_replica.cc:333] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.307097 18904 raft_consensus.cc:2243] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.307395 18904 raft_consensus.cc:2272] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.308176 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 7576613d620548df8dbd661bb74dca45 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.308307 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b50886c57ca84a5ea6c5e5b1278de7ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.309182 18904 ts_tablet_manager.cc:1916] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.309293 18901 ts_tablet_manager.cc:1929] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.309326 18903 raft_consensus.cc:2272] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.309536 18901 log.cc:1199] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b50886c57ca84a5ea6c5e5b1278de7ac
I20260812 20:04:03.310235 18901 ts_tablet_manager.cc:1950] T b50886c57ca84a5ea6c5e5b1278de7ac P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.310707 18903 ts_tablet_manager.cc:1916] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.311573 18901 tablet_replica.cc:333] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.311779 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b50886c57ca84a5ea6c5e5b1278de7ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.311918 18901 raft_consensus.cc:2243] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.312404 18901 raft_consensus.cc:2272] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.317301 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.317873 18901 ts_tablet_manager.cc:1916] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.323593 18904 ts_tablet_manager.cc:1929] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.323993 18904 log.cc:1199] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b50886c57ca84a5ea6c5e5b1278de7ac
I20260812 20:04:03.324965 18904 ts_tablet_manager.cc:1950] T b50886c57ca84a5ea6c5e5b1278de7ac P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.326467 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b50886c57ca84a5ea6c5e5b1278de7ac (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.327041 18904 tablet_replica.cc:333] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.327431 18904 raft_consensus.cc:2243] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.327860 18904 raft_consensus.cc:2272] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.322594 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.014s sys 0.008s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":24157,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":4319,"dirs.run_cpu_time_us":367,"dirs.run_wall_time_us":3952,"lbm_read_time_us":832,"lbm_reads_lt_1ms":29,"lbm_write_time_us":429,"lbm_writes_lt_1ms":9,"mutex_wait_us":34989,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":8371,"threads_started":1,"update_count":33}
I20260812 20:04:03.329107 18904 ts_tablet_manager.cc:1916] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.330376 18903 ts_tablet_manager.cc:1929] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.330636 18903 log.cc:1199] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/4d8b11ecadeb4e25b4eb9abc97920d95
I20260812 20:04:03.331326 18903 ts_tablet_manager.cc:1950] T 4d8b11ecadeb4e25b4eb9abc97920d95 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.332690 18903 tablet_replica.cc:333] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.332866 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.332985 18903 raft_consensus.cc:2243] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.333359 18903 raft_consensus.cc:2272] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.336138 18904 ts_tablet_manager.cc:1929] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.336385 18904 log.cc:1199] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4f48b54fa13d4c1ba18d59e9dd8dbb01
I20260812 20:04:03.337049 18904 ts_tablet_manager.cc:1950] T 4f48b54fa13d4c1ba18d59e9dd8dbb01 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.337055 18901 ts_tablet_manager.cc:1929] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.337343 18901 log.cc:1199] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/af421db565214a209958d4ab416c71e3
I20260812 20:04:03.338114 18901 ts_tablet_manager.cc:1950] T af421db565214a209958d4ab416c71e3 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.338204 18904 tablet_replica.cc:333] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.338387 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 4f48b54fa13d4c1ba18d59e9dd8dbb01 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.338527 18904 raft_consensus.cc:2243] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.338802 18904 raft_consensus.cc:2272] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.339802 18903 ts_tablet_manager.cc:1916] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.340118 18904 ts_tablet_manager.cc:1916] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.340204 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet af421db565214a209958d4ab416c71e3 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.340818 18901 tablet_replica.cc:333] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.341228 18901 raft_consensus.cc:2243] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.341637 18901 raft_consensus.cc:2272] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.342732 18901 ts_tablet_manager.cc:1916] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.349319 18903 ts_tablet_manager.cc:1929] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.349702 18903 log.cc:1199] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5128231e073f4f5c8bd9697a9c23a82f
I20260812 20:04:03.350605 18903 ts_tablet_manager.cc:1950] T 5128231e073f4f5c8bd9697a9c23a82f P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.352077 18903 tablet_replica.cc:333] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.352350 18904 ts_tablet_manager.cc:1929] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.352398 18903 raft_consensus.cc:2243] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.352664 18904 log.cc:1199] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5128231e073f4f5c8bd9697a9c23a82f
I20260812 20:04:03.352813 18903 raft_consensus.cc:2272] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.353421 18904 ts_tablet_manager.cc:1950] T 5128231e073f4f5c8bd9697a9c23a82f P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.354450 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5128231e073f4f5c8bd9697a9c23a82f (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.356068 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.053s user 0.019s sys 0.000s Metrics: {"bytes_written":6937,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1372,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1365,"lbm_writes_lt_1ms":27,"mutex_wait_us":12876,"peak_mem_usage":0,"rows_written":17}
I20260812 20:04:03.357025 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:03.357276 18903 ts_tablet_manager.cc:1916] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.357440 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:03.358129 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20260812 20:04:03.358635 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.359202 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5128231e073f4f5c8bd9697a9c23a82f (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.360198 18904 tablet_replica.cc:333] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.360526 18904 raft_consensus.cc:2243] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.360881 18904 raft_consensus.cc:2272] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.361852 18904 ts_tablet_manager.cc:1916] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.365696 18901 ts_tablet_manager.cc:1929] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:03.366071 18901 log.cc:1199] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/4d8b11ecadeb4e25b4eb9abc97920d95
I20260812 20:04:03.367065 18901 ts_tablet_manager.cc:1950] T 4d8b11ecadeb4e25b4eb9abc97920d95 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.367874 17921 tablet_service.cc:1511] Processing CreateTablet for tablet 76908748ec5d46bc96eed422dc439d09 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.368403 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 76908748ec5d46bc96eed422dc439d09 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.368613 17921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76908748ec5d46bc96eed422dc439d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.369148 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76908748ec5d46bc96eed422dc439d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.369608 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 7e355045a20d491ea86bc9e96c6b8740 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.369799 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.370196 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.370360 17937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e355045a20d491ea86bc9e96c6b8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.370630 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.374297 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.374280 18256 tablet_service.cc:1511] Processing CreateTablet for tablet 76908748ec5d46bc96eed422dc439d09 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.374962 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76908748ec5d46bc96eed422dc439d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.377873 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.379030 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.379565 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.001s sys 0.005s Metrics: {"bytes_written":697,"delete_count":0,"lbm_write_time_us":264,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260812 20:04:03.380139 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260812 20:04:03.380558 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260812 20:04:03.380522 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 7e355045a20d491ea86bc9e96c6b8740 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.381044 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260812 20:04:03.381409 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e355045a20d491ea86bc9e96c6b8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.387564 18909 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:03.390524 18904 ts_tablet_manager.cc:1929] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260812 20:04:03.390949 18904 log.cc:1199] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4d8b11ecadeb4e25b4eb9abc97920d95
I20260812 20:04:03.391878 18904 ts_tablet_manager.cc:1950] T 4d8b11ecadeb4e25b4eb9abc97920d95 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.393533 18904 tablet_replica.cc:333] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.393688 18901 tablet_replica.cc:333] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.394097 18901 raft_consensus.cc:2243] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.394248 18904 raft_consensus.cc:2243] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.394654 18901 raft_consensus.cc:2272] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.394660 18904 raft_consensus.cc:2272] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.394785 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 4d8b11ecadeb4e25b4eb9abc97920d95 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.395841 18901 ts_tablet_manager.cc:1916] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.397392 18904 ts_tablet_manager.cc:1916] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.397435 18247 tablet_service.cc:1511] Processing CreateTablet for tablet 7e355045a20d491ea86bc9e96c6b8740 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.398365 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e355045a20d491ea86bc9e96c6b8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.400538 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.082s user 0.012s sys 0.006s Metrics: {"bytes_written":6937,"cfile_init":1,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":526,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2250,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260812 20:04:03.404184 18249 tablet_service.cc:1511] Processing CreateTablet for tablet 1156cfb6f9804f16bfc82c9f847f865e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.404466 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.404812 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1156cfb6f9804f16bfc82c9f847f865e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.404820 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.405725 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 2b87fff6fdb144df8a25fcd2773503c4 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.406303 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b87fff6fdb144df8a25fcd2773503c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.406560 18242 tablet_service.cc:1511] Processing CreateTablet for tablet 2b87fff6fdb144df8a25fcd2773503c4 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.407281 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b87fff6fdb144df8a25fcd2773503c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.411599 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3945e93f2247639436905ba15d3614 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.412178 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3945e93f2247639436905ba15d3614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.415625 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.194s user 0.007s sys 0.018s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":23833,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":5881,"dirs.run_cpu_time_us":312,"dirs.run_wall_time_us":1377,"lbm_read_time_us":810,"lbm_reads_lt_1ms":23,"lbm_write_time_us":685,"lbm_writes_lt_1ms":9,"mutex_wait_us":11659,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":50150,"threads_started":1,"update_count":33}
I20260812 20:04:03.416898 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.012s sys 0.004s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1677,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":3834,"lbm_read_time_us":540,"lbm_reads_lt_1ms":21,"lbm_write_time_us":501,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260812 20:04:03.404857 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.419315 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.419754 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.419823 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.228784
I20260812 20:04:03.420874 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.350667
I20260812 20:04:03.421748 17920 tablet_service.cc:1511] Processing CreateTablet for tablet 1156cfb6f9804f16bfc82c9f847f865e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.422474 18106 tablet_service.cc:1511] Processing CreateTablet for tablet 4f94b76c5a944b219d310823711ede0c (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.423163 18106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f94b76c5a944b219d310823711ede0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.424558 17920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1156cfb6f9804f16bfc82c9f847f865e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.431298 17927 tablet_service.cc:1511] Processing CreateTablet for tablet 2b87fff6fdb144df8a25fcd2773503c4 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.431981 17927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b87fff6fdb144df8a25fcd2773503c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.435546 17924 tablet_service.cc:1511] Processing CreateTablet for tablet 4f94b76c5a944b219d310823711ede0c (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.436014 17922 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3945e93f2247639436905ba15d3614 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.437435 17922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3945e93f2247639436905ba15d3614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.438158 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260812 20:04:03.439167 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20260812 20:04:03.440266 17938 tablet_service.cc:1511] Processing CreateTablet for tablet 07195ab3590848f0b690bdef9da8d678 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.440918 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07195ab3590848f0b690bdef9da8d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.441604 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 1156cfb6f9804f16bfc82c9f847f865e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.442181 17924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f94b76c5a944b219d310823711ede0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.442540 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1156cfb6f9804f16bfc82c9f847f865e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 20:04:03.443276 17884 consensus_peers.cc:597] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 -> Peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Couldn't send request to peer d222b63b24184ca6bc530a1547fe3297. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5128231e073f4f5c8bd9697a9c23a82f. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:03.449663 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 07195ab3590848f0b690bdef9da8d678 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.450699 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07195ab3590848f0b690bdef9da8d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.452450 18257 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3945e93f2247639436905ba15d3614 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.453146 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3945e93f2247639436905ba15d3614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.463546 18903 ts_tablet_manager.cc:1929] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.464071 18903 log.cc:1199] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/9590af0a4c9543bfb8f07837c115ef60
I20260812 20:04:03.464828 18903 ts_tablet_manager.cc:1950] T 9590af0a4c9543bfb8f07837c115ef60 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.466753 18249 tablet_service.cc:1511] Processing CreateTablet for tablet 07195ab3590848f0b690bdef9da8d678 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.467031 18244 tablet_service.cc:1511] Processing CreateTablet for tablet 4f94b76c5a944b219d310823711ede0c (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.467667 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07195ab3590848f0b690bdef9da8d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.467336 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 9590af0a4c9543bfb8f07837c115ef60 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.469137 18244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f94b76c5a944b219d310823711ede0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.469414 18903 tablet_replica.cc:333] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.470908 18903 raft_consensus.cc:2243] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.471374 18903 raft_consensus.cc:2272] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.472357 17926 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5c3a4033e049de8455913ff24409ce (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.473043 17926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5c3a4033e049de8455913ff24409ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.473059 18903 ts_tablet_manager.cc:1916] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.480759 18481 consensus_queue.cc:1048] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:03.481983 17929 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb3920af5b948ae99c958920784ad1e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.482789 17929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3920af5b948ae99c958920784ad1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.486342 18910 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.487049 18247 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5c3a4033e049de8455913ff24409ce (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.488498 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.069s user 0.002s sys 0.005s Metrics: {"bytes_written":697,"delete_count":0,"lbm_write_time_us":179,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260812 20:04:03.489517 18240 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3c83a10a3540cb8df9de654712b2a3 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.490356 17932 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3c83a10a3540cb8df9de654712b2a3 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.491032 17932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c83a10a3540cb8df9de654712b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.493666 18247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5c3a4033e049de8455913ff24409ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.494669 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c83a10a3540cb8df9de654712b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.499244 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5c3a4033e049de8455913ff24409ce (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.500253 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3c83a10a3540cb8df9de654712b2a3 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.500949 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c83a10a3540cb8df9de654712b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.506707 18091 tablet_service.cc:1511] Processing CreateTablet for tablet c1b9f2328bd3475f8e03aaed2c9374a0 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.510164 18242 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb3920af5b948ae99c958920784ad1e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.512907 18481 consensus_queue.cc:1048] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260812 20:04:03.509023 18105 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb3920af5b948ae99c958920784ad1e (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.514995 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5c3a4033e049de8455913ff24409ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.515329 17935 tablet_service.cc:1511] Processing CreateTablet for tablet 566eee89ac0f466297d3c5c0e4b03f42 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.515882 17935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566eee89ac0f466297d3c5c0e4b03f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.516387 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 566eee89ac0f466297d3c5c0e4b03f42 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.517254 18240 tablet_service.cc:1511] Processing CreateTablet for tablet 0d34e99c1a44421bb8c2a3532a333118 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.517567 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 0d34e99c1a44421bb8c2a3532a333118 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.518383 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3920af5b948ae99c958920784ad1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.519678 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b9f2328bd3475f8e03aaed2c9374a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.523196 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d34e99c1a44421bb8c2a3532a333118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.526044 18100 tablet_service.cc:1511] Processing CreateTablet for tablet 90783e51bc074e2ba59dc73283c0c528 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.527730 17933 tablet_service.cc:1511] Processing CreateTablet for tablet 0d34e99c1a44421bb8c2a3532a333118 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.528112 18089 tablet_service.cc:1511] Processing CreateTablet for tablet a0674220847a41d3969b6f395af86b0f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.528348 17933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d34e99c1a44421bb8c2a3532a333118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.528777 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566eee89ac0f466297d3c5c0e4b03f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.531247 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0674220847a41d3969b6f395af86b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.531999 18100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90783e51bc074e2ba59dc73283c0c528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.537387 18910 tablet_bootstrap.cc:654] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.537804 17925 tablet_service.cc:1511] Processing CreateTablet for tablet 90783e51bc074e2ba59dc73283c0c528 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.538563 17925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90783e51bc074e2ba59dc73283c0c528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.541396 17936 tablet_service.cc:1511] Processing CreateTablet for tablet a0674220847a41d3969b6f395af86b0f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.541788 18104 tablet_service.cc:1511] Processing CreateTablet for tablet a31d93928a3141229557fc9334acb19f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.541992 17936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0674220847a41d3969b6f395af86b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.542392 18104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31d93928a3141229557fc9334acb19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.543423 17928 tablet_service.cc:1511] Processing CreateTablet for tablet a31d93928a3141229557fc9334acb19f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.543929 17928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31d93928a3141229557fc9334acb19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.504405 17931 tablet_service.cc:1511] Processing CreateTablet for tablet c1b9f2328bd3475f8e03aaed2c9374a0 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.546296 17931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b9f2328bd3475f8e03aaed2c9374a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.549585 18249 tablet_service.cc:1511] Processing CreateTablet for tablet a31d93928a3141229557fc9334acb19f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.554486 18904 ts_tablet_manager.cc:1929] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.558501 18256 tablet_service.cc:1511] Processing CreateTablet for tablet c1b9f2328bd3475f8e03aaed2c9374a0 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.554637 18909 tablet_bootstrap.cc:654] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.559207 18240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d34e99c1a44421bb8c2a3532a333118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.559123 18257 tablet_service.cc:1511] Processing CreateTablet for tablet 566eee89ac0f466297d3c5c0e4b03f42 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.559688 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566eee89ac0f466297d3c5c0e4b03f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.559746 18250 tablet_service.cc:1511] Processing CreateTablet for tablet 90783e51bc074e2ba59dc73283c0c528 (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.560206 18250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90783e51bc074e2ba59dc73283c0c528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.565416 18254 tablet_service.cc:1511] Processing CreateTablet for tablet a0674220847a41d3969b6f395af86b0f (DEFAULT_TABLE table=table-4 [id=9ca906da220346789172f86a9efbe043]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260812 20:04:03.565662 18256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b9f2328bd3475f8e03aaed2c9374a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.565954 18254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0674220847a41d3969b6f395af86b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.567772 18242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3920af5b948ae99c958920784ad1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.580879 18249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31d93928a3141229557fc9334acb19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 20:04:03.591722 18903 ts_tablet_manager.cc:1929] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.591912 18903 log.cc:1199] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/de0311de0f984639b73eddfef7076109
I20260812 20:04:03.592607 18903 ts_tablet_manager.cc:1950] T de0311de0f984639b73eddfef7076109 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.593927 18903 tablet_replica.cc:333] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.594269 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet de0311de0f984639b73eddfef7076109 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.595877 18903 raft_consensus.cc:2243] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.604265 18904 log.cc:1199] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/9590af0a4c9543bfb8f07837c115ef60
I20260812 20:04:03.608830 18903 raft_consensus.cc:2272] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.612296 18903 ts_tablet_manager.cc:1916] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.627728 18904 ts_tablet_manager.cc:1950] T 9590af0a4c9543bfb8f07837c115ef60 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.629580 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 9590af0a4c9543bfb8f07837c115ef60 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.632455 18911 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:03.646188 18904 tablet_replica.cc:333] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.646675 18904 raft_consensus.cc:2243] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.647289 18904 raft_consensus.cc:2272] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.649587 18904 ts_tablet_manager.cc:1916] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.654121 18910 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.654497 18910 ts_tablet_manager.cc:1403] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.168s user 0.006s sys 0.008s
I20260812 20:04:03.656006 18910 raft_consensus.cc:359] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.656455 18910 raft_consensus.cc:385] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.656731 18910 raft_consensus.cc:740] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.657413 18910 consensus_queue.cc:260] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.662832 18901 ts_tablet_manager.cc:1929] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.664104 18909 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:03.664474 18909 ts_tablet_manager.cc:1403] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.277s user 0.010s sys 0.000s
I20260812 20:04:03.664767 18910 ts_tablet_manager.cc:1434] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20260812 20:04:03.668097 18909 raft_consensus.cc:359] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.668411 18909 raft_consensus.cc:385] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.668679 18909 raft_consensus.cc:740] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.670499 18910 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.674130 18910 tablet_bootstrap.cc:654] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.674911 18901 log.cc:1199] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/de0311de0f984639b73eddfef7076109
I20260812 20:04:03.669056 18909 consensus_queue.cc:260] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.678323 18909 ts_tablet_manager.cc:1434] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260812 20:04:03.680230 18909 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:03.684031 18909 tablet_bootstrap.cc:654] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.684130 18911 tablet_bootstrap.cc:654] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.685719 18901 ts_tablet_manager.cc:1950] T de0311de0f984639b73eddfef7076109 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.687093 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet de0311de0f984639b73eddfef7076109 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.688956 18903 ts_tablet_manager.cc:1929] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.689816 18901 tablet_replica.cc:333] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.690142 18901 raft_consensus.cc:2243] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.692351 18910 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.692679 18910 ts_tablet_manager.cc:1403] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20260812 20:04:03.694193 18910 raft_consensus.cc:359] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.694459 18910 raft_consensus.cc:385] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.694552 18910 raft_consensus.cc:740] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.694957 18910 consensus_queue.cc:260] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.696951 18910 ts_tablet_manager.cc:1434] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 20:04:03.697902 18910 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.703406 18903 log.cc:1199] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/af421db565214a209958d4ab416c71e3
I20260812 20:04:03.704276 18901 raft_consensus.cc:2272] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.706756 18910 tablet_bootstrap.cc:654] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.709503 18901 ts_tablet_manager.cc:1916] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.711733 18909 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:03.712070 18909 ts_tablet_manager.cc:1403] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.016s
I20260812 20:04:03.712421 18724 raft_consensus.cc:493] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.712666 18838 raft_consensus.cc:493] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.712914 18838 raft_consensus.cc:515] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.712684 18724 raft_consensus.cc:515] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.714184 18909 raft_consensus.cc:359] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.714596 18909 raft_consensus.cc:385] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.714768 18909 raft_consensus.cc:740] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.714933 18724 leader_election.cc:290] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:03.715306 18909 consensus_queue.cc:260] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.716161 18909 ts_tablet_manager.cc:1434] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:04:03.716599 18909 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:03.717183 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76908748ec5d46bc96eed422dc439d09" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:03.717496 18125 raft_consensus.cc:2468] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:03.718122 17669 tablet.cc:2106] T 00000000000000000000000000000000 P 62a275fb48f042dda1fbd49607d55fdb: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260812 20:04:03.719353 18904 ts_tablet_manager.cc:1929] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.719534 18904 log.cc:1199] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/f922ded62e854f609412e537f21eb9b1
I20260812 20:04:03.719540 18208 leader_election.cc:304] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:03.720166 18904 ts_tablet_manager.cc:1950] T f922ded62e854f609412e537f21eb9b1 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.720568 18838 leader_election.cc:290] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:03.720844 18838 raft_consensus.cc:2804] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:03.720989 18838 raft_consensus.cc:493] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:03.721139 18838 raft_consensus.cc:3060] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.724542 18909 tablet_bootstrap.cc:654] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.726336 18911 tablet_bootstrap.cc:492] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:03.726608 18911 ts_tablet_manager.cc:1403] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.000s
I20260812 20:04:03.727258 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.293s user 0.016s sys 0.042s Metrics: {"dirs.queue_time_us":9389,"dirs.run_cpu_time_us":1119,"dirs.run_wall_time_us":13391,"lbm_write_time_us":719,"lbm_writes_lt_1ms":7,"mutex_wait_us":8379,"num_input_rowsets":0,"thread_start_us":246153,"threads_started":1}
I20260812 20:04:03.727546 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:03.728422 18911 raft_consensus.cc:359] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.728704 18911 raft_consensus.cc:385] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.728897 18911 raft_consensus.cc:740] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.729308 18911 consensus_queue.cc:260] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.730268 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5372 bytes on disk
I20260812 20:04:03.730577 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5372 bytes on disk
I20260812 20:04:03.731024 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5372 bytes on disk
I20260812 20:04:03.732014 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20260812 20:04:03.732502 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.732869 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.734439 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:03.734553 17961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76908748ec5d46bc96eed422dc439d09" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:03.734758 17961 raft_consensus.cc:2468] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:03.736699 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet f922ded62e854f609412e537f21eb9b1 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.738024 18904 tablet_replica.cc:333] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.738363 18904 raft_consensus.cc:2243] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.738653 18904 raft_consensus.cc:2272] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.742034 18903 ts_tablet_manager.cc:1950] T af421db565214a209958d4ab416c71e3 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
W20260812 20:04:03.742973 17883 consensus_peers.cc:597] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5128231e073f4f5c8bd9697a9c23a82f. This is attempt 1: this message will repeat every 5th retry.
W20260812 20:04:03.743283 18208 leader_election.cc:343] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:03.743543 18207 leader_election.cc:343] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.743685 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet af421db565214a209958d4ab416c71e3 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.743721 18207 leader_election.cc:304] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:03.743939 18909 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:03.744122 18724 raft_consensus.cc:2749] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:03.744185 18909 ts_tablet_manager.cc:1403] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20260812 20:04:03.744861 18903 tablet_replica.cc:333] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.745463 18909 raft_consensus.cc:359] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.745838 18909 raft_consensus.cc:385] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.746351 18911 ts_tablet_manager.cc:1434] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20260812 20:04:03.746515 18909 raft_consensus.cc:740] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.746925 18909 consensus_queue.cc:260] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.745350 18903 raft_consensus.cc:2243] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.747632 18903 raft_consensus.cc:2272] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.748611 18903 ts_tablet_manager.cc:1916] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.752908 18838 raft_consensus.cc:515] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.754534 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76908748ec5d46bc96eed422dc439d09" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:03.754920 18911 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:03.754889 18125 raft_consensus.cc:3060] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.756348 18909 ts_tablet_manager.cc:1434] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20260812 20:04:03.758103 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76908748ec5d46bc96eed422dc439d09" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:03.758504 17962 raft_consensus.cc:3060] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.761243 18911 tablet_bootstrap.cc:654] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.763440 17962 raft_consensus.cc:2468] T 76908748ec5d46bc96eed422dc439d09 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:03.764005 18838 leader_election.cc:290] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:03.764264 18207 leader_election.cc:304] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:03.764875 18910 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.765148 18910 ts_tablet_manager.cc:1403] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.008s
I20260812 20:04:03.766312 18125 raft_consensus.cc:2468] T 76908748ec5d46bc96eed422dc439d09 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:03.767134 18910 raft_consensus.cc:359] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.767454 18910 raft_consensus.cc:385] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.767683 18910 raft_consensus.cc:740] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.768126 18910 consensus_queue.cc:260] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.768649 18724 raft_consensus.cc:2804] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:03.768863 18724 raft_consensus.cc:697] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:03.769328 18724 consensus_queue.cc:237] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } }
I20260812 20:04:03.773671 18909 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:03.776396 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.347s user 0.041s sys 0.017s Metrics: {"bytes_written":38202,"cfile_cache_hit":13,"cfile_cache_hit_bytes":27163,"cfile_cache_miss":35,"cfile_cache_miss_bytes":39075,"cfile_init":9,"delta_iterators_relevant":7,"dirs.queue_time_us":4254,"dirs.run_cpu_time_us":1258,"dirs.run_wall_time_us":11293,"drs_written":1,"lbm_read_time_us":2147,"lbm_reads_lt_1ms":71,"lbm_write_time_us":1906,"lbm_writes_lt_1ms":28,"mutex_wait_us":284,"num_input_rowsets":2,"peak_mem_usage":153059,"rows_written":88,"thread_start_us":45751,"threads_started":2}
I20260812 20:04:03.777194 18909 tablet_bootstrap.cc:654] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.776451 17770 catalog_manager.cc:5719] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:03.777551 18901 ts_tablet_manager.cc:1929] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.777841 18901 log.cc:1199] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/f922ded62e854f609412e537f21eb9b1
I20260812 20:04:03.778393 18901 ts_tablet_manager.cc:1950] T f922ded62e854f609412e537f21eb9b1 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.779611 18901 tablet_replica.cc:333] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.779898 18901 raft_consensus.cc:2243] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.779928 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet f922ded62e854f609412e537f21eb9b1 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.780095 18901 raft_consensus.cc:2272] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.780562 18903 ts_tablet_manager.cc:1929] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.780853 18903 log.cc:1199] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/f922ded62e854f609412e537f21eb9b1
I20260812 20:04:03.781070 18904 ts_tablet_manager.cc:1916] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.781560 18903 ts_tablet_manager.cc:1950] T f922ded62e854f609412e537f21eb9b1 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.781597 18911 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:03.781927 18911 ts_tablet_manager.cc:1403] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.005s
I20260812 20:04:03.783079 18903 tablet_replica.cc:333] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.783313 18911 raft_consensus.cc:359] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.783399 18903 raft_consensus.cc:2243] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.783655 18903 raft_consensus.cc:2272] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.783669 18911 raft_consensus.cc:385] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.783833 18911 raft_consensus.cc:740] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.784305 18910 ts_tablet_manager.cc:1434] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260812 20:04:03.784781 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet f922ded62e854f609412e537f21eb9b1 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.784906 18903 ts_tablet_manager.cc:1916] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.787528 18910 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.788584 18481 raft_consensus.cc:493] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.788895 18481 raft_consensus.cc:515] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.790001 18481 leader_election.cc:290] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.790381 18910 tablet_bootstrap.cc:654] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.791316 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:03.791378 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:03.791918 18911 consensus_queue.cc:260] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
W20260812 20:04:03.792354 18043 leader_election.cc:343] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:03.792757 18043 leader_election.cc:343] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.793052 18043 leader_election.cc:304] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:03.795485 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.796053 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.796293 18903 ts_tablet_manager.cc:1929] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.796446 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.796478 18903 log.cc:1199] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/2b5eebe47827446aae93c30cdaa4e345
I20260812 20:04:03.796470 18481 raft_consensus.cc:2749] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:03.796810 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.797183 18903 ts_tablet_manager.cc:1950] T 2b5eebe47827446aae93c30cdaa4e345 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.798266 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.798702 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.799168 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.799270 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.799491 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.799746 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.799746 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.800128 18911 ts_tablet_manager.cc:1434] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.018s user 0.001s sys 0.003s
I20260812 20:04:03.800769 18901 ts_tablet_manager.cc:1916] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.801200 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.801589 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.802207 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.804100 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 2b5eebe47827446aae93c30cdaa4e345 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.804913 18903 tablet_replica.cc:333] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.805239 18903 raft_consensus.cc:2243] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.805660 18903 raft_consensus.cc:2272] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.807215 18903 ts_tablet_manager.cc:1916] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 20:04:03.809177 18207 consensus_peers.cc:597] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 -> Peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Couldn't send request to peer f20226c1597d481a84f0c2fe53f0bb65. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:03.810159 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.001s sys 0.006s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":219,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.812282 18911 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:03.813088 18860 raft_consensus.cc:493] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.813300 18860 raft_consensus.cc:515] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.812877 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":218,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.814200 18860 leader_election.cc:290] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.815574 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:03.816313 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:04:03.816529 17884 leader_election.cc:343] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.816622 18281 raft_consensus.cc:2468] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:03.817505 17883 leader_election.cc:304] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:03.818094 18860 raft_consensus.cc:2804] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:03.818477 18860 raft_consensus.cc:493] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:03.818784 18860 raft_consensus.cc:3060] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.819326 18904 ts_tablet_manager.cc:1929] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.819585 18904 log.cc:1199] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/af421db565214a209958d4ab416c71e3
I20260812 20:04:03.822376 18911 tablet_bootstrap.cc:654] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.824710 18910 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.825168 18910 ts_tablet_manager.cc:1403] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.002s
I20260812 20:04:03.826347 18904 ts_tablet_manager.cc:1950] T af421db565214a209958d4ab416c71e3 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.826557 18910 raft_consensus.cc:359] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.823508 18860 raft_consensus.cc:515] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.827966 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet af421db565214a209958d4ab416c71e3 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.828629 18904 tablet_replica.cc:333] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.828905 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:03.829033 18904 raft_consensus.cc:2243] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.830088 18904 raft_consensus.cc:2272] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 20:04:03.830292 17884 leader_election.cc:343] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.830919 18860 leader_election.cc:290] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.832077 18904 ts_tablet_manager.cc:1916] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.835568 18909 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:03.835916 18909 ts_tablet_manager.cc:1403] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.000s
I20260812 20:04:03.837280 18909 raft_consensus.cc:359] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.837643 18909 raft_consensus.cc:385] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.837770 18909 raft_consensus.cc:740] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.838284 18909 consensus_queue.cc:260] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.838975 18909 ts_tablet_manager.cc:1434] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:04:03.842541 18910 raft_consensus.cc:385] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.842718 18910 raft_consensus.cc:740] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.826700 18903 ts_tablet_manager.cc:1929] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.829591 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1156cfb6f9804f16bfc82c9f847f865e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:03.843258 18903 log.cc:1199] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/4581755097204339b4b41750e54ac482
I20260812 20:04:03.843796 18909 tablet_bootstrap.cc:492] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:03.843905 18901 ts_tablet_manager.cc:1929] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.844259 18901 log.cc:1199] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5743e1151ef0487ab67f6478b6126279
I20260812 20:04:03.845568 18901 ts_tablet_manager.cc:1950] T 5743e1151ef0487ab67f6478b6126279 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.845966 18281 raft_consensus.cc:3060] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.847194 18901 tablet_replica.cc:333] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.847479 18901 raft_consensus.cc:2243] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.848138 18911 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:03.848237 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5743e1151ef0487ab67f6478b6126279 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.848665 18911 ts_tablet_manager.cc:1403] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.002s
I20260812 20:04:03.848680 18903 ts_tablet_manager.cc:1950] T 4581755097204339b4b41750e54ac482 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.849849 18281 raft_consensus.cc:2468] T 1156cfb6f9804f16bfc82c9f847f865e P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:03.850178 18909 tablet_bootstrap.cc:654] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.850244 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 4581755097204339b4b41750e54ac482 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.850478 17883 leader_election.cc:304] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters: d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:03.850658 18903 tablet_replica.cc:333] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.850575 18911 raft_consensus.cc:359] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.850991 18903 raft_consensus.cc:2243] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.851064 18860 raft_consensus.cc:2804] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:03.851141 18911 raft_consensus.cc:385] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.851217 18860 raft_consensus.cc:697] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:03.851491 18911 raft_consensus.cc:740] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.851577 18903 raft_consensus.cc:2272] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.851651 18860 consensus_queue.cc:237] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.851922 18911 consensus_queue.cc:260] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.852833 18903 ts_tablet_manager.cc:1916] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.853505 18911 ts_tablet_manager.cc:1434] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260812 20:04:03.854000 18911 tablet_bootstrap.cc:492] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:03.847848 18901 raft_consensus.cc:2272] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.856169 18901 ts_tablet_manager.cc:1916] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.858497 18910 consensus_queue.cc:260] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.859426 18910 ts_tablet_manager.cc:1434] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.034s user 0.000s sys 0.002s
I20260812 20:04:03.861918 18904 ts_tablet_manager.cc:1929] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.862171 18904 log.cc:1199] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/de0311de0f984639b73eddfef7076109
I20260812 20:04:03.862828 18904 ts_tablet_manager.cc:1950] T de0311de0f984639b73eddfef7076109 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.864090 18904 tablet_replica.cc:333] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:03.864246 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet de0311de0f984639b73eddfef7076109 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.864428 18904 raft_consensus.cc:2243] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.864560 18903 ts_tablet_manager.cc:1929] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.864701 18904 raft_consensus.cc:2272] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.864895 18903 log.cc:1199] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/b6d9104457094da480d79c4e41fe9f77
I20260812 20:04:03.866019 18910 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.868384 18904 ts_tablet_manager.cc:1916] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.869071 18910 tablet_bootstrap.cc:654] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.869546 18911 tablet_bootstrap.cc:654] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.870651 17767 catalog_manager.cc:5719] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:03.871527 18903 ts_tablet_manager.cc:1950] T b6d9104457094da480d79c4e41fe9f77 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.872236 18814 raft_consensus.cc:493] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.872488 18814 raft_consensus.cc:515] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.873610 18814 leader_election.cc:290] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.874018 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet b6d9104457094da480d79c4e41fe9f77 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.874995 18903 tablet_replica.cc:333] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.875659 18903 raft_consensus.cc:2243] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.876955 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3945e93f2247639436905ba15d3614" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:03.877364 18125 raft_consensus.cc:2468] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:03.878213 17884 leader_election.cc:304] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:03.884217 18814 raft_consensus.cc:2804] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:03.886245 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3945e93f2247639436905ba15d3614" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:03.886930 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.887369 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.887562 18903 raft_consensus.cc:2272] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.888155 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.888377 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.888505 18903 ts_tablet_manager.cc:1916] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.888715 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.889295 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.890761 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.891258 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.891659 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.892354 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.892572 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.892758 18814 raft_consensus.cc:493] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:03.892988 18814 raft_consensus.cc:3060] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.893137 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260812 20:04:03.893992 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260812 20:04:03.895159 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.895567 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.896344 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260812 20:04:03.905565 17883 leader_election.cc:343] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.907481 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.006s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":270,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.908102 18814 raft_consensus.cc:515] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.911219 18904 ts_tablet_manager.cc:1929] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.911270 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3945e93f2247639436905ba15d3614" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:03.911243 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3945e93f2247639436905ba15d3614" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:03.911517 18904 log.cc:1199] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/568e19899d4140ac8f708e183f2cd8c2
W20260812 20:04:03.912283 17883 leader_election.cc:343] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.912601 18814 leader_election.cc:290] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.912635 18904 ts_tablet_manager.cc:1950] T 568e19899d4140ac8f708e183f2cd8c2 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:03.914091 18125 raft_consensus.cc:3060] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.914984 18911 tablet_bootstrap.cc:492] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:03.915262 18911 ts_tablet_manager.cc:1403] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.015s
I20260812 20:04:03.915731 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 568e19899d4140ac8f708e183f2cd8c2 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.916492 18911 raft_consensus.cc:359] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.916790 18911 raft_consensus.cc:385] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.916828 18901 ts_tablet_manager.cc:1929] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.916976 18901 log.cc:1199] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/9590af0a4c9543bfb8f07837c115ef60
I20260812 20:04:03.917668 18901 ts_tablet_manager.cc:1950] T 9590af0a4c9543bfb8f07837c115ef60 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.918617 18910 tablet_bootstrap.cc:492] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.918898 18901 tablet_replica.cc:333] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.918860 18910 ts_tablet_manager.cc:1403] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.006s
I20260812 20:04:03.919174 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 9590af0a4c9543bfb8f07837c115ef60 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.919195 18901 raft_consensus.cc:2243] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:03.916950 18911 raft_consensus.cc:740] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.920413 18910 raft_consensus.cc:359] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.920605 18911 consensus_queue.cc:260] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.920747 18910 raft_consensus.cc:385] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.920928 18910 raft_consensus.cc:740] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.921460 18910 consensus_queue.cc:260] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.924266 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.035s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":303,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:03.928115 18910 ts_tablet_manager.cc:1434] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260812 20:04:03.929771 18910 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.933494 18911 ts_tablet_manager.cc:1434] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.018s user 0.002s sys 0.002s
I20260812 20:04:03.934370 18911 tablet_bootstrap.cc:492] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:03.935937 18903 ts_tablet_manager.cc:1929] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.936156 18903 log.cc:1199] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/5743e1151ef0487ab67f6478b6126279
I20260812 20:04:03.936825 18903 ts_tablet_manager.cc:1950] T 5743e1151ef0487ab67f6478b6126279 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:03.937703 18910 tablet_bootstrap.cc:654] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.938212 18911 tablet_bootstrap.cc:654] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.939000 18125 raft_consensus.cc:2468] T 5a3945e93f2247639436905ba15d3614 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:03.939812 18901 raft_consensus.cc:2272] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.940719 17884 leader_election.cc:304] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:04:03.941535 18814 raft_consensus.cc:2804] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:03.941815 18814 raft_consensus.cc:697] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:03.942397 18814 consensus_queue.cc:237] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.940878 18901 ts_tablet_manager.cc:1916] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.944728 18904 ts_tablet_manager.cc:1916] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.946055 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 5743e1151ef0487ab67f6478b6126279 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.947947 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.008s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":72,"lbm_read_time_us":258,"lbm_reads_lt_1ms":8,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"mutex_wait_us":30833,"reinsert_count":0,"update_count":2}
I20260812 20:04:03.951032 18903 tablet_replica.cc:333] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:03.951519 18903 raft_consensus.cc:2243] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.951884 18903 raft_consensus.cc:2272] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.953131 18903 ts_tablet_manager.cc:1916] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.957453 18904 ts_tablet_manager.cc:1929] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 20:04:03.958053 17767 catalog_manager.cc:5719] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:03.972515 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.972993 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.973353 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.965159 18481 raft_consensus.cc:493] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:03.974277 18481 raft_consensus.cc:515] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.975396 18481 leader_election.cc:290] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:03.975980 18904 log.cc:1199] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/4581755097204339b4b41750e54ac482
I20260812 20:04:03.976178 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:03.976444 18281 raft_consensus.cc:2468] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 0.
I20260812 20:04:03.977073 18043 leader_election.cc:304] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:03.977586 18481 raft_consensus.cc:2804] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:03.977785 18481 raft_consensus.cc:493] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:03.978044 18481 raft_consensus.cc:3060] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:03.978941 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:03.979269 18910 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:03.979558 18910 ts_tablet_manager.cc:1403] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.004s
W20260812 20:04:03.979710 18043 leader_election.cc:343] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:03.980687 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.980989 18910 raft_consensus.cc:359] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.981370 18910 raft_consensus.cc:385] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.981457 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.981551 18910 raft_consensus.cc:740] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.981873 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.982174 18901 ts_tablet_manager.cc:1929] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.982095 18910 consensus_queue.cc:260] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.982314 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:03.982425 18901 log.cc:1199] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/5128231e073f4f5c8bd9697a9c23a82f
I20260812 20:04:03.982975 18901 ts_tablet_manager.cc:1950] T 5128231e073f4f5c8bd9697a9c23a82f P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:03.984823 18910 ts_tablet_manager.cc:1434] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 20:04:03.985293 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.985433 18910 tablet_bootstrap.cc:492] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:03.985809 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.986259 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.987391 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.987861 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.988356 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:03.989032 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.989318 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:03.990139 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 5128231e073f4f5c8bd9697a9c23a82f (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:03.991030 18910 tablet_bootstrap.cc:654] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:03.992880 18901 tablet_replica.cc:333] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:03.993163 18901 raft_consensus.cc:2243] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:03.993434 18901 raft_consensus.cc:2272] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:03.994439 18901 ts_tablet_manager.cc:1916] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:03.994731 18909 tablet_bootstrap.cc:492] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:03.994977 18909 ts_tablet_manager.cc:1403] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.016s
I20260812 20:04:03.996361 18909 raft_consensus.cc:359] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.996781 18909 raft_consensus.cc:385] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:03.997035 18909 raft_consensus.cc:740] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:03.997243 18481 raft_consensus.cc:515] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.998066 18903 ts_tablet_manager.cc:1929] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:03.997582 18909 consensus_queue.cc:260] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:03.999732 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:04.000643 18909 ts_tablet_manager.cc:1434] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 20:04:04.001085 18909 tablet_bootstrap.cc:492] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.001336 18481 leader_election.cc:290] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.006363 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:04.006649 18281 raft_consensus.cc:3060] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.009056 18901 ts_tablet_manager.cc:1929] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.009286 18901 log.cc:1199] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/b6d9104457094da480d79c4e41fe9f77
W20260812 20:04:04.003607 18043 leader_election.cc:343] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.010471 18901 ts_tablet_manager.cc:1950] T b6d9104457094da480d79c4e41fe9f77 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:04.011482 18281 raft_consensus.cc:2468] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d222b63b24184ca6bc530a1547fe3297 in term 1.
I20260812 20:04:04.012521 18043 leader_election.cc:304] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.013165 18811 raft_consensus.cc:2804] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.013182 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet b6d9104457094da480d79c4e41fe9f77 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.013479 18811 raft_consensus.cc:697] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Becoming Leader. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Running, Role: LEADER
I20260812 20:04:04.013458 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.031s user 0.006s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":260,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.013770 18901 tablet_replica.cc:333] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:04.013900 18811 consensus_queue.cc:237] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.014228 18901 raft_consensus.cc:2243] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:04.014811 18901 raft_consensus.cc:2272] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.015784 18901 ts_tablet_manager.cc:1916] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.016191 18904 ts_tablet_manager.cc:1950] T 4581755097204339b4b41750e54ac482 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:04.018299 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 4581755097204339b4b41750e54ac482 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.006098 18909 tablet_bootstrap.cc:654] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.024075 18911 tablet_bootstrap.cc:492] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.024365 18911 ts_tablet_manager.cc:1403] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.005s
I20260812 20:04:04.025045 18904 tablet_replica.cc:333] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:04.025478 18904 raft_consensus.cc:2243] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:04.025893 18911 raft_consensus.cc:359] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.026230 18911 raft_consensus.cc:385] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.026435 18911 raft_consensus.cc:740] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.027097 18904 raft_consensus.cc:2272] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.027674 18903 log.cc:1199] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/568e19899d4140ac8f708e183f2cd8c2
I20260812 20:04:04.028232 18911 consensus_queue.cc:260] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.035329 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.046s user 0.010s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":716,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":14309,"lbm_read_time_us":319,"lbm_reads_lt_1ms":8,"lbm_write_time_us":284,"lbm_writes_lt_1ms":4,"mutex_wait_us":12672,"reinsert_count":0,"update_count":3}
I20260812 20:04:04.037418 18903 ts_tablet_manager.cc:1950] T 568e19899d4140ac8f708e183f2cd8c2 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:04.038175 18911 ts_tablet_manager.cc:1434] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260812 20:04:04.038877 18901 ts_tablet_manager.cc:1929] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.039196 18901 log.cc:1199] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/568e19899d4140ac8f708e183f2cd8c2
I20260812 20:04:04.039773 18901 ts_tablet_manager.cc:1950] T 568e19899d4140ac8f708e183f2cd8c2 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:04.039973 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 568e19899d4140ac8f708e183f2cd8c2 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.041100 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 568e19899d4140ac8f708e183f2cd8c2 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.041687 18901 tablet_replica.cc:333] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:04.041992 18903 tablet_replica.cc:333] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: stopping tablet replica
I20260812 20:04:04.042002 18901 raft_consensus.cc:2243] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:04.042270 18903 raft_consensus.cc:2243] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:04.042344 18901 raft_consensus.cc:2272] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.043066 17767 catalog_manager.cc:5719] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 reported cstate change: term changed from 0 to 1, leader changed from <none> to d222b63b24184ca6bc530a1547fe3297 (127.25.14.2). New cstate: current_term: 1 leader_uuid: "d222b63b24184ca6bc530a1547fe3297" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:04.045015 18901 ts_tablet_manager.cc:1916] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.046300 18911 tablet_bootstrap.cc:492] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.047770 18903 raft_consensus.cc:2272] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.049196 18903 ts_tablet_manager.cc:1916] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.057880 18911 tablet_bootstrap.cc:654] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.059279 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.086s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":205,"lbm_writes_lt_1ms":4,"mutex_wait_us":63372,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.061194 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.167s user 0.013s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5401,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":28635,"dirs.run_cpu_time_us":426,"dirs.run_wall_time_us":2837,"lbm_read_time_us":1731,"lbm_reads_lt_1ms":27,"lbm_write_time_us":427,"lbm_writes_lt_1ms":9,"mutex_wait_us":13455,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":70251,"threads_started":1,"update_count":2}
I20260812 20:04:04.062212 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.209725
I20260812 20:04:04.062661 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.209725
I20260812 20:04:04.067579 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.069490 18901 ts_tablet_manager.cc:1929] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.069975 18901 log.cc:1199] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/2b5eebe47827446aae93c30cdaa4e345
I20260812 20:04:04.070741 18901 ts_tablet_manager.cc:1950] T 2b5eebe47827446aae93c30cdaa4e345 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:04.071048 18910 tablet_bootstrap.cc:492] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
W20260812 20:04:04.072512 17883 consensus_peers.cc:597] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 -> Peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Couldn't send request to peer 2aa0359db6f441889edca36cd80c4058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4581755097204339b4b41750e54ac482. This is attempt 1: this message will repeat every 5th retry.
I20260812 20:04:04.072697 18901 tablet_replica.cc:333] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: stopping tablet replica
I20260812 20:04:04.073066 18910 ts_tablet_manager.cc:1403] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.007s
I20260812 20:04:04.073104 18901 raft_consensus.cc:2243] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Raft consensus shutting down.
I20260812 20:04:04.061100 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.074501 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.074961 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.075793 18910 raft_consensus.cc:359] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.077366 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.077751 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.078459 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.078786 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.079071 18901 raft_consensus.cc:2272] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.079111 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.079705 18910 raft_consensus.cc:385] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.079845 18910 raft_consensus.cc:740] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.069075 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.095s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":168,"lbm_writes_lt_1ms":4,"mutex_wait_us":78204,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.069571 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.095s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":95301}
I20260812 20:04:04.073076 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 2b5eebe47827446aae93c30cdaa4e345 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.080590 18901 ts_tablet_manager.cc:1916] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.081147 18911 tablet_bootstrap.cc:492] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.081300 18910 consensus_queue.cc:260] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.081488 18911 ts_tablet_manager.cc:1403] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.006s
I20260812 20:04:04.082649 18911 raft_consensus.cc:359] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.083644 18903 ts_tablet_manager.cc:1929] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.083837 18903 log.cc:1199] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-2/wal/wals/7576613d620548df8dbd661bb74dca45
I20260812 20:04:04.084194 18910 ts_tablet_manager.cc:1434] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20260812 20:04:04.084590 18910 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:04.082855 18911 raft_consensus.cc:385] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.085551 18911 raft_consensus.cc:740] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.088792 18903 ts_tablet_manager.cc:1950] T 7576613d620548df8dbd661bb74dca45 P 2aa0359db6f441889edca36cd80c4058: Deleting consensus metadata
I20260812 20:04:04.089094 18911 consensus_queue.cc:260] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.090381 17750 catalog_manager.cc:5050] TS 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): tablet 7576613d620548df8dbd661bb74dca45 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.092293 18911 ts_tablet_manager.cc:1434] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260812 20:04:04.094501 18838 raft_consensus.cc:493] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.094797 18838 raft_consensus.cc:515] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.095590 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.003s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":218,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.095865 18838 leader_election.cc:290] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.097069 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:04:04.097877 18207 leader_election.cc:343] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.097831 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.098248 18125 raft_consensus.cc:2468] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:04.098590 18910 tablet_bootstrap.cc:654] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.098968 18911 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.099217 18208 leader_election.cc:304] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.099645 18838 raft_consensus.cc:2804] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:04.099891 18838 raft_consensus.cc:493] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:04.100049 18838 raft_consensus.cc:3060] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.101643 18901 ts_tablet_manager.cc:1929] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.101886 18901 log.cc:1199] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-0/wal/wals/4581755097204339b4b41750e54ac482
I20260812 20:04:04.102208 18911 tablet_bootstrap.cc:654] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.102748 18901 ts_tablet_manager.cc:1950] T 4581755097204339b4b41750e54ac482 P f20226c1597d481a84f0c2fe53f0bb65: Deleting consensus metadata
I20260812 20:04:04.103453 18904 ts_tablet_manager.cc:1916] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.104609 17750 catalog_manager.cc:5050] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): tablet 4581755097204339b4b41750e54ac482 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.104631 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.007s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":739,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1378,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":681,"lbm_read_time_us":184,"lbm_reads_lt_1ms":8,"lbm_write_time_us":236,"lbm_writes_lt_1ms":4,"mutex_wait_us":5266,"reinsert_count":0,"update_count":4}
W20260812 20:04:04.105103 17750 catalog_manager.cc:5017] TS f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): delete failed for tablet 7576613d620548df8dbd661bb74dca45 because the tablet was not found. No further retry: Not found: Tablet not found: 7576613d620548df8dbd661bb74dca45
I20260812 20:04:04.106191 18909 tablet_bootstrap.cc:492] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.106077 18838 raft_consensus.cc:515] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.106530 18909 ts_tablet_manager.cc:1403] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.105s user 0.000s sys 0.011s
I20260812 20:04:04.107566 18909 raft_consensus.cc:359] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.107916 18909 raft_consensus.cc:385] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.107923 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:04.108072 18909 raft_consensus.cc:740] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.108482 18909 consensus_queue.cc:260] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.109056 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:04.109345 18125 raft_consensus.cc:3060] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:04.111274 18207 leader_election.cc:343] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.111707 18838 leader_election.cc:290] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.113374 18125 raft_consensus.cc:2468] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:04.114269 18208 leader_election.cc:304] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters: f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.114928 18838 raft_consensus.cc:2804] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.109323 18909 ts_tablet_manager.cc:1434] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:04:04.116269 18909 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.116726 18838 raft_consensus.cc:697] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:04.117043 18911 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.117278 18911 ts_tablet_manager.cc:1403] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260812 20:04:04.117197 18838 consensus_queue.cc:237] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.118774 18911 raft_consensus.cc:359] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.119009 18911 raft_consensus.cc:385] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.119287 18911 raft_consensus.cc:740] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.119699 18911 consensus_queue.cc:260] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.121536 18909 tablet_bootstrap.cc:654] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.125124 18911 ts_tablet_manager.cc:1434] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.008s user 0.000s sys 0.004s
I20260812 20:04:04.129480 18911 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.132887 18911 tablet_bootstrap.cc:654] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.139648 18911 tablet_bootstrap.cc:492] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.139909 18911 ts_tablet_manager.cc:1403] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260812 20:04:04.141019 18911 raft_consensus.cc:359] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.141566 18911 raft_consensus.cc:385] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.141681 18911 raft_consensus.cc:740] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.141985 18911 consensus_queue.cc:260] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.142844 18911 ts_tablet_manager.cc:1434] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:04:04.143363 18911 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.144369 17767 catalog_manager.cc:5719] T 7e355045a20d491ea86bc9e96c6b8740 P 2aa0359db6f441889edca36cd80c4058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2aa0359db6f441889edca36cd80c4058 (127.25.14.3). New cstate: current_term: 1 leader_uuid: "2aa0359db6f441889edca36cd80c4058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: HEALTHY } } }
I20260812 20:04:04.146680 18911 tablet_bootstrap.cc:654] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.154313 18838 raft_consensus.cc:493] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.154732 18838 raft_consensus.cc:515] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.156543 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3c83a10a3540cb8df9de654712b2a3" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:04.157169 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3c83a10a3540cb8df9de654712b2a3" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:04.158372 18208 leader_election.cc:343] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.158954 18838 leader_election.cc:290] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.160282 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.160663 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.161006 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.161393 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
W20260812 20:04:04.161536 18207 leader_election.cc:343] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.161913 18207 leader_election.cc:304] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.164069 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.164462 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.164507 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.164903 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.164871 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.164983 18838 raft_consensus.cc:2749] T 4c3c83a10a3540cb8df9de654712b2a3 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.165223 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.165664 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.166023 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.166347 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.166730 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.167212 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.180049 18911 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.180331 18911 ts_tablet_manager.cc:1403] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.023s
I20260812 20:04:04.181737 18911 raft_consensus.cc:359] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.182085 18911 raft_consensus.cc:385] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.182266 18911 raft_consensus.cc:740] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.182723 18909 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.182675 18911 consensus_queue.cc:260] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.182978 18909 ts_tablet_manager.cc:1403] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.006s
I20260812 20:04:04.183385 18911 ts_tablet_manager.cc:1434] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 20:04:04.183897 18911 tablet_bootstrap.cc:492] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.184386 18909 raft_consensus.cc:359] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.184664 18909 raft_consensus.cc:385] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.184854 18909 raft_consensus.cc:740] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.185457 18909 consensus_queue.cc:260] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.187204 18909 ts_tablet_manager.cc:1434] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260812 20:04:04.187386 18911 tablet_bootstrap.cc:654] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.187614 18909 tablet_bootstrap.cc:492] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.188833 18838 raft_consensus.cc:493] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.189178 18838 raft_consensus.cc:515] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.189774 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":235,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.190264 18838 leader_election.cc:290] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.191298 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07195ab3590848f0b690bdef9da8d678" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:04.192512 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07195ab3590848f0b690bdef9da8d678" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
W20260812 20:04:04.192638 18207 leader_election.cc:343] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:04.193464 18208 leader_election.cc:343] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.193766 18208 leader_election.cc:304] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058; no voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.197324 18838 raft_consensus.cc:2749] T 07195ab3590848f0b690bdef9da8d678 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.199407 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.033s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":767,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":90,"lbm_read_time_us":236,"lbm_reads_lt_1ms":8,"lbm_write_time_us":223,"lbm_writes_lt_1ms":4,"mutex_wait_us":6074,"reinsert_count":0,"update_count":5}
I20260812 20:04:04.201179 18910 tablet_bootstrap.cc:492] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:04.201671 18910 ts_tablet_manager.cc:1403] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.019s
I20260812 20:04:04.202957 18904 ts_tablet_manager.cc:1929] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.202883 18910 raft_consensus.cc:359] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.203143 18910 raft_consensus.cc:385] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.203233 18910 raft_consensus.cc:740] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.203697 18910 consensus_queue.cc:260] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.208535 18904 log.cc:1199] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/2b5eebe47827446aae93c30cdaa4e345
I20260812 20:04:04.209395 18904 ts_tablet_manager.cc:1950] T 2b5eebe47827446aae93c30cdaa4e345 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:04.210117 18910 ts_tablet_manager.cc:1434] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260812 20:04:04.204645 18909 tablet_bootstrap.cc:654] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.213351 18910 tablet_bootstrap.cc:492] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:04.214830 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 2b5eebe47827446aae93c30cdaa4e345 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.218052 18910 tablet_bootstrap.cc:654] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.218509 18904 tablet_replica.cc:333] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:04.219151 18904 raft_consensus.cc:2243] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:04.220438 18904 raft_consensus.cc:2272] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.222620 18904 ts_tablet_manager.cc:1916] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.228910 18910 tablet_bootstrap.cc:492] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:04.229215 18910 ts_tablet_manager.cc:1403] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260812 20:04:04.230515 18909 tablet_bootstrap.cc:492] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.230885 18909 ts_tablet_manager.cc:1403] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.003s
I20260812 20:04:04.231083 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.065s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":65274}
I20260812 20:04:04.231173 17755 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.065s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":65145}
I20260812 20:04:04.231595 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.066s user 0.008s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":290,"lbm_writes_lt_1ms":4,"mutex_wait_us":56456,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.232734 18909 raft_consensus.cc:359] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.232731 18910 raft_consensus.cc:359] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.232947 18910 raft_consensus.cc:385] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.233219 18909 raft_consensus.cc:385] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.233382 18909 raft_consensus.cc:740] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.233357 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.169s user 0.016s sys 0.017s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5475,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":4648,"dirs.run_cpu_time_us":441,"dirs.run_wall_time_us":6488,"lbm_read_time_us":1497,"lbm_reads_lt_1ms":31,"lbm_write_time_us":545,"lbm_writes_lt_1ms":9,"mutex_wait_us":6674,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":69632,"threads_started":1,"update_count":2}
I20260812 20:04:04.233808 18909 consensus_queue.cc:260] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.233966 18838 consensus_queue.cc:1048] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:04.234395 18909 ts_tablet_manager.cc:1434] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 20:04:04.233106 18910 raft_consensus.cc:740] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.236063 18910 consensus_queue.cc:260] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.238842 18909 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.243731 18911 tablet_bootstrap.cc:492] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.244966 18911 ts_tablet_manager.cc:1403] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.061s user 0.020s sys 0.032s
I20260812 20:04:04.246105 18909 tablet_bootstrap.cc:654] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.246623 18911 raft_consensus.cc:359] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.248216 18838 consensus_queue.cc:1048] T 76908748ec5d46bc96eed422dc439d09 P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:04.248520 18911 raft_consensus.cc:385] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.249157 18911 raft_consensus.cc:740] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.249897 18911 consensus_queue.cc:260] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.251464 18911 ts_tablet_manager.cc:1434] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 20:04:04.251906 18814 raft_consensus.cc:493] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.252691 18814 raft_consensus.cc:515] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.253722 18814 leader_election.cc:290] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.254190 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0674220847a41d3969b6f395af86b0f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.254669 18911 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.254666 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0674220847a41d3969b6f395af86b0f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.257330 18910 ts_tablet_manager.cc:1434] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
W20260812 20:04:04.255038 17884 leader_election.cc:343] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:04.258176 17883 leader_election.cc:343] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.258764 17883 leader_election.cc:304] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:04.260180 18814 raft_consensus.cc:2749] T a0674220847a41d3969b6f395af86b0f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.260730 18904 ts_tablet_manager.cc:1929] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.263530 18904 log.cc:1199] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/b6d9104457094da480d79c4e41fe9f77
I20260812 20:04:04.264644 18904 ts_tablet_manager.cc:1950] T b6d9104457094da480d79c4e41fe9f77 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:04.266343 18814 raft_consensus.cc:493] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.266654 18814 raft_consensus.cc:515] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.267594 18814 leader_election.cc:290] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.268113 18911 tablet_bootstrap.cc:654] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.268599 18909 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.268908 18909 ts_tablet_manager.cc:1403] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20260812 20:04:04.269676 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a31d93928a3141229557fc9334acb19f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.270056 18124 raft_consensus.cc:2468] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:04.270342 18909 raft_consensus.cc:359] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.270640 18909 raft_consensus.cc:385] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.270862 18909 raft_consensus.cc:740] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.271366 18909 consensus_queue.cc:260] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.272107 18909 ts_tablet_manager.cc:1434] T 4c3c83a10a3540cb8df9de654712b2a3 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 20:04:04.272148 17884 leader_election.cc:304] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.272608 18909 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.272953 18814 raft_consensus.cc:2804] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:04.273136 18814 raft_consensus.cc:493] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:04.273388 18814 raft_consensus.cc:3060] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.274125 18904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 129) took 1057 ms. Trace:
I20260812 20:04:04.274322 18904 rpcz_store.cc:276] 0812 20:04:03.216790 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:03.218075 (+ 1285us) service_pool.cc:224] Handling call
0812 20:04:04.218315 (+1000240us) ts_tablet_manager.cc:1151] Deleting tablet b6d9104457094da480d79c4e41fe9f77
0812 20:04:04.218342 (+ 27us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:04.246828 (+ 28486us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:04.260610 (+ 13782us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:04.273972 (+ 13362us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999889}
I20260812 20:04:04.275288 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet b6d9104457094da480d79c4e41fe9f77 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.279482 18909 tablet_bootstrap.cc:654] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.280915 18904 tablet_replica.cc:333] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: stopping tablet replica
I20260812 20:04:04.283396 18904 raft_consensus.cc:2243] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 20:04:04.284235 18904 raft_consensus.cc:2272] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 20:04:04.284188 18814 raft_consensus.cc:515] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.284592 18911 tablet_bootstrap.cc:492] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.284926 18911 ts_tablet_manager.cc:1403] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.007s
I20260812 20:04:04.285310 18904 ts_tablet_manager.cc:1916] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 20:04:04.285431 18814 leader_election.cc:290] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.287106 18911 raft_consensus.cc:359] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.287533 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a31d93928a3141229557fc9334acb19f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:04.287793 18911 raft_consensus.cc:385] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.287891 18124 raft_consensus.cc:3060] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.288594 18911 raft_consensus.cc:740] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.289285 18911 consensus_queue.cc:260] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.290086 18911 ts_tablet_manager.cc:1434] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 20:04:04.290671 18911 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.292984 18911 tablet_bootstrap.cc:654] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.268687 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a31d93928a3141229557fc9334acb19f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.286505 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a31d93928a3141229557fc9334acb19f" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:04.294171 18910 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
W20260812 20:04:04.295300 17883 leader_election.cc:343] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:04.296000 17883 leader_election.cc:343] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.297652 18909 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.298149 18909 ts_tablet_manager.cc:1403] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.007s
I20260812 20:04:04.299861 18910 tablet_bootstrap.cc:654] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.299785 18909 raft_consensus.cc:359] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.300280 18909 raft_consensus.cc:385] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.300504 18909 raft_consensus.cc:740] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.300940 18909 consensus_queue.cc:260] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.301731 18909 ts_tablet_manager.cc:1434] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260812 20:04:04.302333 18909 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.303817 18904 ts_tablet_manager.cc:1929] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260812 20:04:04.305909 18904 log.cc:1199] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Deleting WAL directory at /tmp/dist-test-taskQcgiFK/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1786564362379865-25656-0/minicluster-data/ts-1/wal/wals/5743e1151ef0487ab67f6478b6126279
I20260812 20:04:04.306620 18904 ts_tablet_manager.cc:1950] T 5743e1151ef0487ab67f6478b6126279 P d222b63b24184ca6bc530a1547fe3297: Deleting consensus metadata
I20260812 20:04:04.309716 18904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 128) took 1093 ms. Trace:
I20260812 20:04:04.310129 18904 rpcz_store.cc:276] 0812 20:04:03.216617 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:03.218171 (+ 1554us) service_pool.cc:224] Handling call
0812 20:04:04.280558 (+1062387us) ts_tablet_manager.cc:1151] Deleting tablet 5743e1151ef0487ab67f6478b6126279
0812 20:04:04.280574 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:04.290303 (+ 9729us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:04.303699 (+ 13396us) tablet_metadata.cc:665] Metadata flushed
0812 20:04:04.309666 (+ 5967us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062024}
I20260812 20:04:04.310765 18811 raft_consensus.cc:493] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.310883 18124 raft_consensus.cc:2468] T a31d93928a3141229557fc9334acb19f P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:04.311133 17751 catalog_manager.cc:5050] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): tablet 5743e1151ef0487ab67f6478b6126279 (table table-3 [id=eef07a79cfc64743a1848fd8c3d42f4f]) successfully deleted
I20260812 20:04:04.311061 18811 raft_consensus.cc:515] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.308576 18909 tablet_bootstrap.cc:654] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.311491 17884 leader_election.cc:304] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058
I20260812 20:04:04.311930 18814 raft_consensus.cc:2804] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.312171 18811 leader_election.cc:290] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.312294 18814 raft_consensus.cc:697] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:04.312793 18814 consensus_queue.cc:237] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.313582 18904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40402 (request call id 132) took 1094 ms. Trace:
I20260812 20:04:04.313915 18904 rpcz_store.cc:276] 0812 20:04:03.219189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0812 20:04:03.219904 (+ 715us) service_pool.cc:224] Handling call
0812 20:04:04.313303 (+1093399us) ts_tablet_manager.cc:1151] Deleting tablet 7576613d620548df8dbd661bb74dca45
0812 20:04:04.313315 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0812 20:04:04.313474 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092974}
W20260812 20:04:04.314684 17751 catalog_manager.cc:5017] TS d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): delete failed for tablet 7576613d620548df8dbd661bb74dca45 because the tablet was not found. No further retry: Not found: Tablet not found: 7576613d620548df8dbd661bb74dca45
I20260812 20:04:04.315943 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566eee89ac0f466297d3c5c0e4b03f42" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
W20260812 20:04:04.317142 18043 leader_election.cc:343] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.318382 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566eee89ac0f466297d3c5c0e4b03f42" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
W20260812 20:04:04.319681 18043 leader_election.cc:343] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.320030 18043 leader_election.cc:304] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.320631 18811 raft_consensus.cc:2749] T 566eee89ac0f466297d3c5c0e4b03f42 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.331920 18909 tablet_bootstrap.cc:492] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.332211 18909 ts_tablet_manager.cc:1403] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.009s
I20260812 20:04:04.333889 18909 raft_consensus.cc:359] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.334197 18909 raft_consensus.cc:385] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.334467 18909 raft_consensus.cc:740] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.334882 18909 consensus_queue.cc:260] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.335649 18909 ts_tablet_manager.cc:1434] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:04:04.336545 18909 tablet_bootstrap.cc:492] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.338346 18860 raft_consensus.cc:493] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.338672 18860 raft_consensus.cc:515] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.339911 18860 leader_election.cc:290] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.340143 18909 tablet_bootstrap.cc:654] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.341367 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.342926 18277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b87fff6fdb144df8a25fcd2773503c4" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.343351 18277 raft_consensus.cc:2393] T 2b87fff6fdb144df8a25fcd2773503c4 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate d222b63b24184ca6bc530a1547fe3297 in this term.
I20260812 20:04:04.344003 17883 leader_election.cc:304] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:04.344390 18911 tablet_bootstrap.cc:492] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65: No bootstrap required, opened a new log
I20260812 20:04:04.344553 18860 raft_consensus.cc:3060] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.344738 18911 ts_tablet_manager.cc:1403] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent bootstrapping tablet: real 0.054s user 0.024s sys 0.026s
I20260812 20:04:04.347123 18911 raft_consensus.cc:359] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.347477 18911 raft_consensus.cc:385] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.347649 18911 raft_consensus.cc:740] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.348073 18911 consensus_queue.cc:260] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.348775 18911 ts_tablet_manager.cc:1434] T 4c3c83a10a3540cb8df9de654712b2a3 P f20226c1597d481a84f0c2fe53f0bb65: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 20:04:04.349256 18911 tablet_bootstrap.cc:492] T 0d34e99c1a44421bb8c2a3532a333118 P f20226c1597d481a84f0c2fe53f0bb65: Bootstrap starting.
I20260812 20:04:04.349995 18860 raft_consensus.cc:2749] T 2b87fff6fdb144df8a25fcd2773503c4 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.350899 18277 raft_consensus.cc:3060] T 5a3945e93f2247639436905ba15d3614 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.351277 18910 tablet_bootstrap.cc:492] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058: No bootstrap required, opened a new log
I20260812 20:04:04.351647 18910 ts_tablet_manager.cc:1403] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.028s
I20260812 20:04:04.352833 18911 tablet_bootstrap.cc:654] T 0d34e99c1a44421bb8c2a3532a333118 P f20226c1597d481a84f0c2fe53f0bb65: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.353574 18910 raft_consensus.cc:359] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.355432 18909 tablet_bootstrap.cc:492] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.355695 18909 ts_tablet_manager.cc:1403] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20260812 20:04:04.356771 18909 raft_consensus.cc:359] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.357077 18909 raft_consensus.cc:385] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.357229 18909 raft_consensus.cc:740] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.357800 18909 consensus_queue.cc:260] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.359917 18909 ts_tablet_manager.cc:1434] T a0674220847a41d3969b6f395af86b0f P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260812 20:04:04.360669 18909 tablet_bootstrap.cc:492] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.360481 18860 consensus_queue.cc:1048] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:04.364478 18909 tablet_bootstrap.cc:654] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.364428 18713 raft_consensus.cc:493] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.364812 18713 raft_consensus.cc:515] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.366245 18713 leader_election.cc:290] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.372256 17767 catalog_manager.cc:5719] T a31d93928a3141229557fc9334acb19f P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260812 20:04:04.388393 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.388832 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.389154 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.389832 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.390082 18909 tablet_bootstrap.cc:492] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.390302 18909 ts_tablet_manager.cc:1403] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.006s
I20260812 20:04:04.390831 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.391391 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.391865 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.392300 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260812 20:04:04.392648 17670 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.393275 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90783e51bc074e2ba59dc73283c0c528" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.393246 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.393646 18124 raft_consensus.cc:2468] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:04.393720 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.394582 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.394939 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.395171 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.207045
I20260812 20:04:04.395460 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.395671 17759 maintenance_manager.cc:419] P a3cb6cd752da451f99894b4d53d113d1: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.207045
I20260812 20:04:04.396101 17758 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.397603 17884 leader_election.cc:304] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.398253 18811 raft_consensus.cc:493] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.398545 18811 raft_consensus.cc:515] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.398938 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.399849 18811 leader_election.cc:290] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.400600 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d34e99c1a44421bb8c2a3532a333118" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:04.402364 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d34e99c1a44421bb8c2a3532a333118" candidate_uuid: "d222b63b24184ca6bc530a1547fe3297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.402233 18976 consensus_queue.cc:1048] T 5a3945e93f2247639436905ba15d3614 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 20:04:04.404822 18976 raft_consensus.cc:2804] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:04.405241 18976 raft_consensus.cc:493] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:04.405557 18976 raft_consensus.cc:3060] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.404153 18909 raft_consensus.cc:359] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.405973 18909 raft_consensus.cc:385] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.409103 18814 raft_consensus.cc:493] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.409463 18814 raft_consensus.cc:515] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.409854 18976 raft_consensus.cc:515] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.410136 17756 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.005s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":267,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.410596 18909 raft_consensus.cc:740] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
W20260812 20:04:04.411691 18043 leader_election.cc:343] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:04.411966 18043 leader_election.cc:343] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.412160 18043 leader_election.cc:304] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d222b63b24184ca6bc530a1547fe3297; no voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65
I20260812 20:04:04.412617 18811 raft_consensus.cc:2749] T 0d34e99c1a44421bb8c2a3532a333118 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.413264 17671 maintenance_manager.cc:419] P 62a275fb48f042dda1fbd49607d55fdb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260812 20:04:04.413543 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb3920af5b948ae99c958920784ad1e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.413967 17668 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260812 20:04:04.414572 18910 raft_consensus.cc:385] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.414803 18910 raft_consensus.cc:740] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.414716 18909 consensus_queue.cc:260] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.414817 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90783e51bc074e2ba59dc73283c0c528" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.415251 18910 consensus_queue.cc:260] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.415352 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb3920af5b948ae99c958920784ad1e" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.416189 18909 ts_tablet_manager.cc:1434] T 07195ab3590848f0b690bdef9da8d678 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20260812 20:04:04.416270 18279 raft_consensus.cc:2468] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:04.417953 18910 ts_tablet_manager.cc:1434] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058: Time spent starting tablet: real 0.066s user 0.002s sys 0.001s
I20260812 20:04:04.418424 18910 tablet_bootstrap.cc:492] T a0674220847a41d3969b6f395af86b0f P 2aa0359db6f441889edca36cd80c4058: Bootstrap starting.
I20260812 20:04:04.421453 17669 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":245,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260812 20:04:04.422587 17667 maintenance_manager.cc:643] P 62a275fb48f042dda1fbd49607d55fdb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.007s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":801,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":10702,"lbm_read_time_us":217,"lbm_reads_lt_1ms":8,"lbm_write_time_us":219,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":6}
I20260812 20:04:04.423755 18910 tablet_bootstrap.cc:654] T a0674220847a41d3969b6f395af86b0f P 2aa0359db6f441889edca36cd80c4058: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.423532 17757 maintenance_manager.cc:643] P a3cb6cd752da451f99894b4d53d113d1: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.009s sys 0.007s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5549,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":168,"lbm_read_time_us":2021,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":26,"lbm_write_time_us":434,"lbm_writes_lt_1ms":9,"mutex_wait_us":5536,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260812 20:04:04.424357 18909 tablet_bootstrap.cc:492] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.410634 18814 leader_election.cc:290] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.414402 18713 raft_consensus.cc:493] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.427829 18976 leader_election.cc:290] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.428066 18713 raft_consensus.cc:515] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.428818 18909 tablet_bootstrap.cc:654] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.429057 18713 leader_election.cc:290] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
W20260812 20:04:04.417188 17883 leader_election.cc:343] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495): Illegal state: must be running to vote when last-logged opid is not known
W20260812 20:04:04.429811 17884 leader_election.cc:343] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.430197 17884 leader_election.cc:304] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:04.431052 18976 raft_consensus.cc:2749] T 9cb3920af5b948ae99c958920784ad1e P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.431682 18814 raft_consensus.cc:493] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.431994 18814 raft_consensus.cc:515] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.432345 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c5c3a4033e049de8455913ff24409ce" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.432756 18281 raft_consensus.cc:2468] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 0.
I20260812 20:04:04.432833 18814 leader_election.cc:290] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.433427 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90783e51bc074e2ba59dc73283c0c528" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:04.433738 18279 raft_consensus.cc:3060] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.434087 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90783e51bc074e2ba59dc73283c0c528" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:04.434406 18125 raft_consensus.cc:3060] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.438560 18127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c5c3a4033e049de8455913ff24409ce" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.438745 18126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.439102 17883 leader_election.cc:304] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.439320 18811 consensus_queue.cc:1048] T 2b87fff6fdb144df8a25fcd2773503c4 P d222b63b24184ca6bc530a1547fe3297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 20:04:04.439426 17884 leader_election.cc:343] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.439762 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e355045a20d491ea86bc9e96c6b8740" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058" is_pre_election: true
I20260812 20:04:04.440307 18126 raft_consensus.cc:2393] T 7e355045a20d491ea86bc9e96c6b8740 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20226c1597d481a84f0c2fe53f0bb65 in current term 1: Already voted for candidate 2aa0359db6f441889edca36cd80c4058 in this term.
I20260812 20:04:04.440346 18713 raft_consensus.cc:2804] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:04.440591 18713 raft_consensus.cc:493] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:04.440833 18713 raft_consensus.cc:3060] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.442287 18125 raft_consensus.cc:2468] T 90783e51bc074e2ba59dc73283c0c528 P d222b63b24184ca6bc530a1547fe3297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:04.445254 18713 raft_consensus.cc:515] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.445520 18909 tablet_bootstrap.cc:492] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.446018 18909 ts_tablet_manager.cc:1403] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20260812 20:04:04.447124 18279 raft_consensus.cc:2468] T 90783e51bc074e2ba59dc73283c0c528 P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:04.447638 18909 raft_consensus.cc:359] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.447971 18909 raft_consensus.cc:385] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.448145 18909 raft_consensus.cc:740] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.447974 17883 leader_election.cc:304] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.448740 18909 consensus_queue.cc:260] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.449707 18909 ts_tablet_manager.cc:1434] T c1b9f2328bd3475f8e03aaed2c9374a0 P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 20:04:04.450178 18909 tablet_bootstrap.cc:492] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.450814 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c5c3a4033e049de8455913ff24409ce" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa0359db6f441889edca36cd80c4058"
I20260812 20:04:04.451215 18279 raft_consensus.cc:3060] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.453837 18909 tablet_bootstrap.cc:654] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297: Neither blocks nor log segments found. Creating new log.
I20260812 20:04:04.453935 18838 raft_consensus.cc:493] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 20:04:04.454113 18838 raft_consensus.cc:515] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.454107 18713 leader_election.cc:290] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Requested vote from peers d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129), 2aa0359db6f441889edca36cd80c4058 (127.25.14.3:34495)
I20260812 20:04:04.454806 18713 raft_consensus.cc:2804] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.455017 18838 leader_election.cc:290] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.455255 18713 raft_consensus.cc:697] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:04.455857 18713 consensus_queue.cc:237] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.457055 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f94b76c5a944b219d310823711ede0c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297" is_pre_election: true
I20260812 20:04:04.457456 18125 raft_consensus.cc:2468] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:04.457854 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f94b76c5a944b219d310823711ede0c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65" is_pre_election: true
I20260812 20:04:04.458130 17962 raft_consensus.cc:2468] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 0.
I20260812 20:04:04.458238 18208 leader_election.cc:304] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297; no voters:
I20260812 20:04:04.459432 18838 raft_consensus.cc:2804] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 20:04:04.459616 18838 raft_consensus.cc:493] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 20:04:04.459836 18838 raft_consensus.cc:3060] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.462285 18279 raft_consensus.cc:2468] T 4c5c3a4033e049de8455913ff24409ce P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20226c1597d481a84f0c2fe53f0bb65 in term 1.
I20260812 20:04:04.463768 17883 leader_election.cc:304] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.463922 18838 raft_consensus.cc:515] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.464490 17884 leader_election.cc:304] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f20226c1597d481a84f0c2fe53f0bb65; no voters: 2aa0359db6f441889edca36cd80c4058, d222b63b24184ca6bc530a1547fe3297
I20260812 20:04:04.464797 18976 raft_consensus.cc:2804] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.465046 18860 consensus_queue.cc:1048] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:04.465242 18972 raft_consensus.cc:3060] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.466149 17962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f94b76c5a944b219d310823711ede0c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20226c1597d481a84f0c2fe53f0bb65"
I20260812 20:04:04.466461 17962 raft_consensus.cc:3060] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.467020 18838 leader_election.cc:290] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Requested vote from peers f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1:46337), d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129)
I20260812 20:04:04.469734 18909 tablet_bootstrap.cc:492] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297: No bootstrap required, opened a new log
I20260812 20:04:04.469981 18909 ts_tablet_manager.cc:1403] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20260812 20:04:04.471390 18909 raft_consensus.cc:359] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.471668 18909 raft_consensus.cc:385] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 20:04:04.471887 18909 raft_consensus.cc:740] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222b63b24184ca6bc530a1547fe3297, State: Initialized, Role: FOLLOWER
I20260812 20:04:04.472362 18909 consensus_queue.cc:260] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.473071 18909 ts_tablet_manager.cc:1434] T 9cb3920af5b948ae99c958920784ad1e P d222b63b24184ca6bc530a1547fe3297: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 20:04:04.473430 18909 tablet_bootstrap.cc:492] T 4c5c3a4033e049de8455913ff24409ce P d222b63b24184ca6bc530a1547fe3297: Bootstrap starting.
I20260812 20:04:04.473728 18972 raft_consensus.cc:2749] T 7e355045a20d491ea86bc9e96c6b8740 P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 20:04:04.473965 18125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c5c3a4033e049de8455913ff24409ce" candidate_uuid: "f20226c1597d481a84f0c2fe53f0bb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:04.474596 18126 raft_consensus.cc:3060] T 1156cfb6f9804f16bfc82c9f847f865e P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
W20260812 20:04:04.476482 17884 leader_election.cc:343] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d222b63b24184ca6bc530a1547fe3297 (127.25.14.2:34129): Illegal state: must be running to vote when last-logged opid is not known
I20260812 20:04:04.477993 17962 raft_consensus.cc:2468] T 4f94b76c5a944b219d310823711ede0c P f20226c1597d481a84f0c2fe53f0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aa0359db6f441889edca36cd80c4058 in term 1.
I20260812 20:04:04.478957 18207 leader_election.cc:304] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa0359db6f441889edca36cd80c4058, f20226c1597d481a84f0c2fe53f0bb65; no voters:
I20260812 20:04:04.479493 18838 raft_consensus.cc:2804] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 20:04:04.479763 18838 raft_consensus.cc:697] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa0359db6f441889edca36cd80c4058, State: Running, Role: LEADER
I20260812 20:04:04.480255 18838 consensus_queue.cc:237] T 4f94b76c5a944b219d310823711ede0c P 2aa0359db6f441889edca36cd80c4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.482861 18972 consensus_queue.cc:1048] T 1156cfb6f9804f16bfc82c9f847f865e P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 20:04:04.487783 18124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f94b76c5a944b219d310823711ede0c" candidate_uuid: "2aa0359db6f441889edca36cd80c4058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d222b63b24184ca6bc530a1547fe3297"
I20260812 20:04:04.488307 18124 raft_consensus.cc:3060] T 4f94b76c5a944b219d310823711ede0c P d222b63b24184ca6bc530a1547fe3297 [term 0 FOLLOWER]: Advancing to term 1
I20260812 20:04:04.490635 18976 raft_consensus.cc:697] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [term 1 LEADER]: Becoming Leader. State: Replica: f20226c1597d481a84f0c2fe53f0bb65, State: Running, Role: LEADER
I20260812 20:04:04.494378 18976 consensus_queue.cc:237] T 4c5c3a4033e049de8455913ff24409ce P f20226c1597d481a84f0c2fe53f0bb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058" member_type: VOTER last_known_addr { host: "127.25.14.3" port: 34495 } }
I20260812 20:04:04.503576 17767 catalog_manager.cc:5719] T 90783e51bc074e2ba59dc73283c0c528 P f20226c1597d481a84f0c2fe53f0bb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20226c1597d481a84f0c2fe53f0bb65 (127.25.14.1). New cstate: current_term: 1 leader_uuid: "f20226c1597d481a84f0c2fe53f0bb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20226c1597d481a84f0c2fe53f0bb65" member_type: VOTER last_known_addr { host: "127.25.14.1" port: 46337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d222b63b24184ca6bc530a1547fe3297" member_type: VOTER last_known_addr { host: "127.25.14.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa0359db6f441889edca36cd80c4058"